[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:39 dist-test-slave-dzwq krb5kdc[3690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:39 dist-test-slave-dzwq krb5kdc[3690](info): set up 2 sockets
Sep 25 10:12:39 dist-test-slave-dzwq krb5kdc[3690](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 10:12:40.076704 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:40.220037 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41495
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:40.308655 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:40.308969 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:40.309021 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:40.310405 3703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:40.310498 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:40.310525 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:40.310565 3703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:40.310627 3703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:40.312516 3703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:40.312863 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:40.313141 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:40.319558 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.319571 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.319571 3703 server_base.cc:1034] running on GCE node
W20260925 10:12:40.319592 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.320242 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:40.320550 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:40.321723 3703 hybrid_clock.cc:648] HybridClock initialized: now 1790331160321698 us; error 44 us; skew 500 ppm
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:40.324734 3703 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:40.325274 3703 webserver.cc:460] Webserver started at http://127.3.152.254:42599/ using document root <none> and password file <none>
I20260925 10:12:40.325526 3703 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:40.325613 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:40.325766 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:40.326921 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "60d7ca0346834780bec89de2416c2e9b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "492321109108303683a2d05ad703e3d5"
server_key_iv: "e43b778abe80c54fb0cedf6b91ac1f9d"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.327319 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "60d7ca0346834780bec89de2416c2e9b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "492321109108303683a2d05ad703e3d5"
server_key_iv: "e43b778abe80c54fb0cedf6b91ac1f9d"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.329015 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:40.330082 3718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.330358 3703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:40.330500 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "60d7ca0346834780bec89de2416c2e9b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "492321109108303683a2d05ad703e3d5"
server_key_iv: "e43b778abe80c54fb0cedf6b91ac1f9d"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.330591 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:40.338999 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:40.341531 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:40.341743 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:40.346356 3703 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41495
I20260925 10:12:40.346395 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41495 every 8 connection(s)
I20260925 10:12:40.346809 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:40.347867 3771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:40.350420 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b: Bootstrap starting.
I20260925 10:12:40.351086 3771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:40.351402 3771 log.cc:824] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:40.352420 3771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b: No bootstrap required, opened a new log
I20260925 10:12:40.354242 3771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } }
I20260925 10:12:40.354403 3771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:40.354429 3771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d7ca0346834780bec89de2416c2e9b, State: Initialized, Role: FOLLOWER
I20260925 10:12:40.354637 3771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } }
I20260925 10:12:40.354714 3771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:40.354740 3771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:40.354799 3771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:40.355471 3771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } }
I20260925 10:12:40.355607 3771 leader_election.cc:302] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60d7ca0346834780bec89de2416c2e9b; no voters:
I20260925 10:12:40.355855 3771 leader_election.cc:288] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:40.355847 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 3703
I20260925 10:12:40.355890 3774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:40.355962 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:40.356034 3774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [term 1 LEADER]: Becoming Leader. State: Replica: 60d7ca0346834780bec89de2416c2e9b, State: Running, Role: LEADER
I20260925 10:12:40.356132 3774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } }
I20260925 10:12:40.356176 3771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:40.356467 3774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60d7ca0346834780bec89de2416c2e9b. Latest consensus state: current_term: 1 leader_uuid: "60d7ca0346834780bec89de2416c2e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } } }
I20260925 10:12:40.356642 3774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:40.356918 3683 external_mini_cluster.cc:1428] Setting key 63090b3abb221a1ca988fa70fd29c9ff
I20260925 10:12:40.356635 3776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60d7ca0346834780bec89de2416c2e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60d7ca0346834780bec89de2416c2e9b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41495 } } }
I20260925 10:12:40.356961 3776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:40.357089 3782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:40.357710 3782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:40.359597 3782 catalog_manager.cc:1347] Generated new cluster ID: 2e2510b88100413b94f89676d38ed24e
I20260925 10:12:40.359669 3782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:40.375236 3782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:40.375993 3782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:40.377404 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.358023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.3.152.254:41495)
0925 10:12:40.358471 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.358482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.358646 (+ 164us) server_negotiation.cc:407] Connection header received
0925 10:12:40.361204 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.361213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.361275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.361411 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.363582 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.365184 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.366751 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.367459 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.367574 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.372632 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.372637 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.372638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.372655 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.374997 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.375646 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.375650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.375651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.375766 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.376169 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.376170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.376171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.376439 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.376607 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.376935 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.377104 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":81,"threads_started":1}
I20260925 10:12:40.387439 3782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60d7ca0346834780bec89de2416c2e9b: Generated new TSK 0
I20260925 10:12:40.387769 3782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:40.428059 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:40.527514 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:40.527760 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:40.527787 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:40.529182 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:40.529232 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:40.529264 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:40.530917 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:40.531216 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:40.531497 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:40.537424 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.537424 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.537429 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.537509 3797 server_base.cc:1034] running on GCE node
I20260925 10:12:40.537995 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:40.538290 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:40.539460 3797 hybrid_clock.cc:648] HybridClock initialized: now 1790331160539437 us; error 43 us; skew 500 ppm
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:40.542484 3797 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:40.543043 3797 webserver.cc:460] Webserver started at http://127.3.152.193:34565/ using document root <none> and password file <none>
I20260925 10:12:40.543319 3797 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:40.543401 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:40.543562 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:40.544937 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "a94773ec54a74cb3b504417b556ee351"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "16a18adf9049126fb258d886e58a1fa7"
server_key_iv: "0d6c8dd3b61554a72fdb893cd930f54c"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.545336 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "a94773ec54a74cb3b504417b556ee351"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "16a18adf9049126fb258d886e58a1fa7"
server_key_iv: "0d6c8dd3b61554a72fdb893cd930f54c"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.546913 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:40.547880 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.548175 3797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:40.548310 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "a94773ec54a74cb3b504417b556ee351"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "16a18adf9049126fb258d886e58a1fa7"
server_key_iv: "0d6c8dd3b61554a72fdb893cd930f54c"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.548409 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:40.557910 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:40.560582 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:40.560783 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:40.561045 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:40.561401 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:40.561460 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.561499 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:40.561551 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.568198 3797 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:33685
I20260925 10:12:40.568290 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:33685 every 8 connection(s)
I20260925 10:12:40.568642 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:40.574493 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 3797
I20260925 10:12:40.574656 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:40.574851 3683 external_mini_cluster.cc:1428] Setting key 3c8ba0f5ba6338459872f2accfa0358d
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:40.580149 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.569545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:38061 (local address 127.3.152.254:41495)
0925 10:12:40.569623 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.569626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.569857 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:12:40.570389 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.570390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.570401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.570464 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.571520 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.572131 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.573360 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.573588 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.573663 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.576390 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.576391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.576391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.576401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.578384 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.578918 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.578920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.578920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.579029 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.579326 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.579327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.579327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.579476 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.579562 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.579892 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.580031 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:12:40.580365 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.569509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41495 (local address 127.3.152.193:38061)
0925 10:12:40.569705 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:40.569753 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:40.570291 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:40.570529 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:40.570532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:40.571184 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:40.571415 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.571421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.572249 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.572250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.573209 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.573212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.573682 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.573683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.573809 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.574543 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:40.574552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:40.576284 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:40.578512 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.578521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.578524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.578813 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.579121 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.579121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.579122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.579236 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.579563 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:40.579573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:40.579731 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:12:40.579851 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:12:40.580083 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260925 10:12:40.580960 3926 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41495
I20260925 10:12:40.581066 3926 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:40.581336 3926 heartbeater.cc:511] Master 127.3.152.254:41495 requested a full tablet report, sending...
I20260925 10:12:40.581986 3735 ts_manager.cc:194] Registered new tserver with Master: a94773ec54a74cb3b504417b556ee351 (127.3.152.193:33685)
I20260925 10:12:40.582726 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:38061
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:40.603785 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:40.686776 3931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:40.687045 3931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:40.687088 3931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:40.688318 3931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:40.688397 3931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:40.688443 3931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:40.689960 3931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:40.690255 3931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:40.690505 3931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:40.695814 3939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.695834 3937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.696019 3931 server_base.cc:1034] running on GCE node
W20260925 10:12:40.695817 3936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.696319 3931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:40.696602 3931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:40.697784 3931 hybrid_clock.cc:648] HybridClock initialized: now 1790331160697794 us; error 68 us; skew 500 ppm
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:40.700489 3931 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:40.700985 3931 webserver.cc:460] Webserver started at http://127.3.152.194:44569/ using document root <none> and password file <none>
I20260925 10:12:40.701251 3931 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:40.701320 3931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:40.701470 3931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:40.702580 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "3e42f64db6294074b6ec442b0bcb70ef"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1846d0d1dc3ff04b951846cb1485910d"
server_key_iv: "7244496c197b7553f97f94e29345cd1b"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.702955 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "3e42f64db6294074b6ec442b0bcb70ef"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1846d0d1dc3ff04b951846cb1485910d"
server_key_iv: "7244496c197b7553f97f94e29345cd1b"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.704370 3931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:40.705132 3946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.705374 3931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:40.705473 3931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "3e42f64db6294074b6ec442b0bcb70ef"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1846d0d1dc3ff04b951846cb1485910d"
server_key_iv: "7244496c197b7553f97f94e29345cd1b"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.705559 3931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:40.718060 3931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:40.720502 3931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:40.720682 3931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:40.720933 3931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:40.721266 3931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:40.721320 3931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.721355 3931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:40.721401 3931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.727272 3931 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35613
I20260925 10:12:40.727365 4060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35613 every 8 connection(s)
I20260925 10:12:40.727702 3931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:40.729458 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 3931
I20260925 10:12:40.729568 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:40.729743 3683 external_mini_cluster.cc:1428] Setting key 326cfafbf615da61bf326ce13eafbb27
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:12:40.738629 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.728609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:48711 (local address 127.3.152.254:41495)
0925 10:12:40.728674 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.728676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.728892 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:12:40.729458 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.729459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.729469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.729557 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.730726 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.731219 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.732505 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.732902 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.732984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.735354 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.735356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.735357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.735371 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.737092 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.737525 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.737527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.737528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.737596 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.737856 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.737857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.737858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.738023 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.738107 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.738385 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.738479 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:12:40.738758 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.728607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41495 (local address 127.3.152.194:48711)
0925 10:12:40.728761 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:40.728799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:40.729345 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:40.729643 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:40.729648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:40.730325 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:40.730609 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.730616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.731319 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.731320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.732415 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.732418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.732993 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.732994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.733115 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.733730 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:40.733739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:40.735253 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:40.737188 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.737200 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.737202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.737436 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.737677 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.737678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.737678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.737784 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.738097 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:40.738101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:40.738250 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 10:12:40.738336 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 10:12:40.738463 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260925 10:12:40.739367 4061 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41495
I20260925 10:12:40.739490 4061 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:40.739682 4061 heartbeater.cc:511] Master 127.3.152.254:41495 requested a full tablet report, sending...
I20260925 10:12:40.740177 3735 ts_manager.cc:194] Registered new tserver with Master: 3e42f64db6294074b6ec442b0bcb70ef (127.3.152.194:35613)
I20260925 10:12:40.740732 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:48711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:40.759140 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:40.840754 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:40.841023 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:40.841064 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:40.842270 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:40.842345 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:40.842392 4066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:40.843900 4066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41495
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:40.844233 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:40.844480 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:40.849714 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.849736 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:40.849776 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:40.849848 4066 server_base.cc:1034] running on GCE node
I20260925 10:12:40.850121 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:40.850383 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:40.851567 4066 hybrid_clock.cc:648] HybridClock initialized: now 1790331160851521 us; error 77 us; skew 500 ppm
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:40.854270 4066 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:40.854677 4066 webserver.cc:460] Webserver started at http://127.3.152.195:37927/ using document root <none> and password file <none>
I20260925 10:12:40.854926 4066 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:40.854974 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:40.855108 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:40.856328 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "152cb336dc4941a7a5f5964aa74f9f8b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1cb12b6daf6596ee50c0222bc86e2be6"
server_key_iv: "28868ca55377918a4b4911becad3a467"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.856688 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "152cb336dc4941a7a5f5964aa74f9f8b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1cb12b6daf6596ee50c0222bc86e2be6"
server_key_iv: "28868ca55377918a4b4911becad3a467"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.858163 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:40.859004 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.859211 4066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:40.859321 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "152cb336dc4941a7a5f5964aa74f9f8b"
format_stamp: "Formatted at 2026-09-25 10:12:40 on dist-test-slave-dzwq"
server_key: "1cb12b6daf6596ee50c0222bc86e2be6"
server_key_iv: "28868ca55377918a4b4911becad3a467"
server_key_version: "encryptionkey@0"
I20260925 10:12:40.859400 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:40.870739 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:40.872931 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:40.873113 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:40.873371 4066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:40.873718 4066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:40.873777 4066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.873822 4066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:40.873852 4066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:40.880117 4066 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:41977
I20260925 10:12:40.880191 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:41977 every 8 connection(s)
I20260925 10:12:40.880558 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:40.884217 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4066
I20260925 10:12:40.884325 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:40.884491 3683 external_mini_cluster.cc:1428] Setting key 369b0147854fbcc47aea0801e24401cc
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:40.890607 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.881436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:41711 (local address 127.3.152.254:41495)
0925 10:12:40.881528 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.881531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.881762 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:12:40.882310 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.882312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.882325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.882397 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.883402 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.884045 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.885320 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.885693 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.885776 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.888007 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.888009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.888010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.888019 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.889422 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.889800 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.889801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.889801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.889858 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.890071 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.890072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.890072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.890184 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.890240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.890471 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.890534 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 10:12:40.890805 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.881388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41495 (local address 127.3.152.195:41711)
0925 10:12:40.881628 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:40.881668 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:40.882171 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:40.882494 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:40.882498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:40.883070 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:40.883278 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.883283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.884138 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.884138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.885223 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:40.885226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.885789 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.885790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:40.885900 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.886466 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:40.886474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:40.887876 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:40.889497 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.889503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.889505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.889728 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.889919 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:40.889919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:40.889920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:40.890017 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:40.890239 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:40.890243 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:40.890360 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 10:12:40.890443 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:12:40.890563 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 10:12:40.891312 4196 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41495
I20260925 10:12:40.891417 4196 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:40.891607 4196 heartbeater.cc:511] Master 127.3.152.254:41495 requested a full tablet report, sending...
I20260925 10:12:40.892022 3735 ts_manager.cc:194] Registered new tserver with Master: 152cb336dc4941a7a5f5964aa74f9f8b (127.3.152.195:41977)
I20260925 10:12:40.892424 3735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:41711
I20260925 10:12:40.896938 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:40.921912 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.913032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.3.152.193:33685)
0925 10:12:40.913189 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.913195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.913212 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:12:40.913263 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.913265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.913281 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.913357 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.913915 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.914784 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.915588 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.916092 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.916156 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.918786 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.918794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.918795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.918803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.920446 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.920895 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.920902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.920902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.920967 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.921250 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.921251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.921251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.921415 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.921606 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.921681 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.921774 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260925 10:12:40.922348 3840 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39682
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:12:40 dist-test-slave-dzwq krb5kdc[3690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:40.947012 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:40.938652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.3.152.193:33685)
0925 10:12:40.938775 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:40.938809 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:40.938927 (+ 118us) server_negotiation.cc:407] Connection header received
0925 10:12:40.939073 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:40.939074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:40.939095 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:40.939217 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:40.939903 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.940482 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.941325 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:40.941549 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:40.941642 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:40.944405 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:40.944407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:40.944407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:40.944415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:40.945895 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.946249 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.946250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.946250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.946310 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:40.946529 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:40.946530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:40.946530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:40.946657 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:40.946774 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:40.946877 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:12:40.946943 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 10:12:40.947669 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 3797
I20260925 10:12:40.952828 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 3931
I20260925 10:12:40.957564 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4066
I20260925 10:12:40.962069 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 3703
2026-09-25T10:12:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1949 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:41 dist-test-slave-dzwq krb5kdc[4210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:41 dist-test-slave-dzwq krb5kdc[4210](info): set up 2 sockets
Sep 25 10:12:41 dist-test-slave-dzwq krb5kdc[4210](info): commencing operation
krb5kdc: starting...
W20260925 10:12:42.004284 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:42.131531 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:40271
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:40271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:42.213949 4223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:42.214217 4223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:42.214254 4223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:42.215466 4223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:42.215553 4223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:42.215577 4223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:42.215597 4223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:42.215629 4223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:42.217274 4223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:40271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:40271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:42.217620 4223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:42.217880 4223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:42.223012 4229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.223137 4223 server_base.cc:1034] running on GCE node
W20260925 10:12:42.223032 4228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:42.223033 4231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.223519 4223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:42.223800 4223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:42.224944 4223 hybrid_clock.cc:648] HybridClock initialized: now 1790331162224920 us; error 47 us; skew 500 ppm
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:42.227824 4223 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:42.228348 4223 webserver.cc:460] Webserver started at http://127.3.152.254:41043/ using document root <none> and password file <none>
I20260925 10:12:42.228595 4223 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:42.228642 4223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:42.228782 4223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:42.229902 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "443351908a7c4ce4b96430e92990bce1"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "335ac42a03c10abc3a19233a783516a3"
server_key_iv: "b8787685764fc6c06f48f4d8bf450fe8"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.230291 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "443351908a7c4ce4b96430e92990bce1"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "335ac42a03c10abc3a19233a783516a3"
server_key_iv: "b8787685764fc6c06f48f4d8bf450fe8"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.231897 4223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:42.232832 4239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.233055 4223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:42.233170 4223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "443351908a7c4ce4b96430e92990bce1"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "335ac42a03c10abc3a19233a783516a3"
server_key_iv: "b8787685764fc6c06f48f4d8bf450fe8"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.233253 4223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:42.251477 4223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:42.253803 4223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:42.253984 4223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:42.258368 4223 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:40271
I20260925 10:12:42.258409 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:40271 every 8 connection(s)
I20260925 10:12:42.258834 4223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:42.259783 4292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:42.262377 4292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1: Bootstrap starting.
I20260925 10:12:42.262946 4292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:42.263197 4292 log.cc:824] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:42.263947 4292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1: No bootstrap required, opened a new log
I20260925 10:12:42.265564 4292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } }
I20260925 10:12:42.265733 4292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:42.265758 4292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443351908a7c4ce4b96430e92990bce1, State: Initialized, Role: FOLLOWER
I20260925 10:12:42.265924 4292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } }
I20260925 10:12:42.266014 4292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:42.266040 4292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:42.266078 4292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:42.266592 4292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } }
I20260925 10:12:42.266677 4292 leader_election.cc:302] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443351908a7c4ce4b96430e92990bce1; no voters:
I20260925 10:12:42.266819 4292 leader_election.cc:288] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:42.266897 4295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:42.267066 4295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [term 1 LEADER]: Becoming Leader. State: Replica: 443351908a7c4ce4b96430e92990bce1, State: Running, Role: LEADER
I20260925 10:12:42.267138 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4223
I20260925 10:12:42.267159 4295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } }
I20260925 10:12:42.267220 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:42.267392 3683 external_mini_cluster.cc:1428] Setting key 1970ee0029eb209610330910521f3c89
I20260925 10:12:42.267535 4297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443351908a7c4ce4b96430e92990bce1. Latest consensus state: current_term: 1 leader_uuid: "443351908a7c4ce4b96430e92990bce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } } }
I20260925 10:12:42.267627 4297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:42.267671 4296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443351908a7c4ce4b96430e92990bce1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443351908a7c4ce4b96430e92990bce1" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 40271 } } }
I20260925 10:12:42.267752 4296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:42.267868 4292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:42.267943 4299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:42.268549 4299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:42.270107 4299 catalog_manager.cc:1347] Generated new cluster ID: 92fd29163e814df3a3084f3b44bfa06a
I20260925 10:12:42.270159 4299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:42.279938 4299 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:42.280723 4299 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:42.282934 4299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443351908a7c4ce4b96430e92990bce1: Generated new TSK 0
I20260925 10:12:42.283100 4299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:42.283360 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.268038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.3.152.254:40271)
0925 10:12:42.268258 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.268268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.269312 (+ 1044us) server_negotiation.cc:407] Connection header received
0925 10:12:42.269522 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.269530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.269600 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.269727 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.272635 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.274143 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.275227 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.275906 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.276107 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.279071 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.279075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.279076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.279091 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.281278 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.281716 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.281719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.281719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.281842 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.282072 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.282073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.282073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.282386 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.282554 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.282934 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.283138 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:42.311570 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:42.396306 4318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:42.396517 4318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:42.396544 4318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:42.397791 4318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:42.397840 4318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:42.397867 4318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:42.399386 4318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:42.399631 4318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:42.399899 4318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:42.405265 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.405262 4318 server_base.cc:1034] running on GCE node
W20260925 10:12:42.405298 4326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:42.405257 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.405788 4318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:42.406003 4318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:42.407146 4318 hybrid_clock.cc:648] HybridClock initialized: now 1790331162407124 us; error 37 us; skew 500 ppm
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:42.409984 4318 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:42.410524 4318 webserver.cc:460] Webserver started at http://127.3.152.193:46019/ using document root <none> and password file <none>
I20260925 10:12:42.410761 4318 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:42.410830 4318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:42.410983 4318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:42.412189 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "9b78dc2d9eed4e179eb1f23a0fefae94"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "3e275b93454f3e1a1ff19c1a845c7517"
server_key_iv: "03ef8a3a33ec90736d01bdb6f726b436"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.412571 4318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "9b78dc2d9eed4e179eb1f23a0fefae94"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "3e275b93454f3e1a1ff19c1a845c7517"
server_key_iv: "03ef8a3a33ec90736d01bdb6f726b436"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.413971 4318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:42.414742 4333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.414948 4318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:42.415055 4318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "9b78dc2d9eed4e179eb1f23a0fefae94"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "3e275b93454f3e1a1ff19c1a845c7517"
server_key_iv: "03ef8a3a33ec90736d01bdb6f726b436"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.415130 4318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:42.428849 4318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:42.431212 4318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:42.431420 4318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:42.431684 4318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:42.432166 4318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:42.432222 4318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.432255 4318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:42.432277 4318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.438463 4318 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:36281
I20260925 10:12:42.438504 4447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:36281 every 8 connection(s)
I20260925 10:12:42.438867 4318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:42.447018 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4318
I20260925 10:12:42.447124 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:42.447312 3683 external_mini_cluster.cc:1428] Setting key 140d71b96f65143035dbb630ae765f3d
I20260925 10:12:42.449148 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.439685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:56985 (local address 127.3.152.254:40271)
0925 10:12:42.439735 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.439737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.439950 (+ 213us) server_negotiation.cc:407] Connection header received
0925 10:12:42.440936 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.440937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.440948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.441004 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.441657 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.442161 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.443163 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.443372 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.443428 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.445846 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.445847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.445848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.445855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.447452 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.448006 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.448008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.448008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.448078 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.448383 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.448384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.448384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.448537 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.448610 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.448888 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.448996 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:42.449147 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.439661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:40271 (local address 127.3.152.193:56985)
0925 10:12:42.439824 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.439862 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.440838 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.441062 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.441066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.441353 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:42.441557 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.441562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.442257 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.442258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.443083 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.443085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.443446 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.443447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.443546 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.444190 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:42.444199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:42.445707 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:42.447542 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.447551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.447554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.447888 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.448195 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.448196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.448197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.448310 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.448603 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:42.448608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:42.448750 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.448840 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.448973 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260925 10:12:42.449734 4448 heartbeater.cc:347] Connected to a master server at 127.3.152.254:40271
I20260925 10:12:42.449837 4448 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:42.450042 4448 heartbeater.cc:511] Master 127.3.152.254:40271 requested a full tablet report, sending...
I20260925 10:12:42.450688 4256 ts_manager.cc:194] Registered new tserver with Master: 9b78dc2d9eed4e179eb1f23a0fefae94 (127.3.152.193:36281)
I20260925 10:12:42.451664 4256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:56985
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:42.475826 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:42.559782 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:42.560060 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:42.560099 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:42.561563 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:42.561697 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:42.561761 4453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:42.563504 4453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:42.563810 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:42.564122 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:42.569517 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:42.569520 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:42.569510 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.569953 4453 server_base.cc:1034] running on GCE node
I20260925 10:12:42.570179 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:42.570474 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:42.571715 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790331162571700 us; error 28 us; skew 500 ppm
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:42.574440 4453 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:42.574980 4453 webserver.cc:460] Webserver started at http://127.3.152.194:43871/ using document root <none> and password file <none>
I20260925 10:12:42.575246 4453 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:42.575294 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:42.575470 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:42.576689 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "248bf3288d094518833c9153a4a96744"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "43b03adddc1d46d54046ebae744b7e3c"
server_key_iv: "bb018a7d5c90ca442f8022bfbff0e533"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.577060 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "248bf3288d094518833c9153a4a96744"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "43b03adddc1d46d54046ebae744b7e3c"
server_key_iv: "bb018a7d5c90ca442f8022bfbff0e533"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.578480 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:42.579334 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.579509 4453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:42.579614 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "248bf3288d094518833c9153a4a96744"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "43b03adddc1d46d54046ebae744b7e3c"
server_key_iv: "bb018a7d5c90ca442f8022bfbff0e533"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.579700 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:42.596285 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:42.598277 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:42.598450 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:42.598703 4453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:42.599023 4453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:42.599086 4453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.599128 4453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:42.599152 4453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.605291 4453 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:33001
I20260925 10:12:42.605340 4582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:33001 every 8 connection(s)
I20260925 10:12:42.605683 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:42.611151 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4453
I20260925 10:12:42.611259 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:42.611452 3683 external_mini_cluster.cc:1428] Setting key 699a10f7f6376cff6a6cc1845e615416
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:42.616506 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.606463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53907 (local address 127.3.152.254:40271)
0925 10:12:42.606550 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.606552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.606772 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:12:42.607775 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.607776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.607789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.607848 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.608550 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.609069 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.610151 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.610371 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.610452 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.613224 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.613225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.613226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.613242 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.614846 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.615258 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.615260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.615260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.615327 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.615689 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.615690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.615690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.615850 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.615927 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.616302 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.616392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:12:42.616667 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.606431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:40271 (local address 127.3.152.194:53907)
0925 10:12:42.606617 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.606658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.607643 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.607934 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.607938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.608224 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:42.608441 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.608446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.609172 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.609173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.610051 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.610054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.610461 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.610463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.610592 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.611219 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:42.611231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:42.613112 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:42.614915 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.614921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.614923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.615164 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.615423 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.615424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.615425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.615542 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.615921 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:42.615952 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:42.616152 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.616258 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.616393 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260925 10:12:42.617254 4583 heartbeater.cc:347] Connected to a master server at 127.3.152.254:40271
I20260925 10:12:42.617393 4583 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:42.617626 4583 heartbeater.cc:511] Master 127.3.152.254:40271 requested a full tablet report, sending...
I20260925 10:12:42.618176 4256 ts_manager.cc:194] Registered new tserver with Master: 248bf3288d094518833c9153a4a96744 (127.3.152.194:33001)
I20260925 10:12:42.618649 4256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:53907
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:42.639786 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:42.722015 4589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:42.722280 4589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:42.722326 4589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:42.723517 4589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:42.723593 4589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:42.723641 4589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:42.725255 4589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:40271
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:42.725551 4589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:42.725798 4589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:42.731051 4595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:42.731038 4597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.731251 4589 server_base.cc:1034] running on GCE node
W20260925 10:12:42.731035 4594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:42.731559 4589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:42.731822 4589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:42.733011 4589 hybrid_clock.cc:648] HybridClock initialized: now 1790331162732986 us; error 51 us; skew 500 ppm
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:42.742255 4589 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:42.742944 4589 webserver.cc:460] Webserver started at http://127.3.152.195:33575/ using document root <none> and password file <none>
I20260925 10:12:42.743233 4589 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:42.743323 4589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:42.743492 4589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:42.744824 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "9213ab98636c426bb4911aa50e9732cd"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "1af123e0a3d1380756cb5a8783f2f3f2"
server_key_iv: "5a378f8f9dbd9740c529bef1b8b9f7c3"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.745255 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "9213ab98636c426bb4911aa50e9732cd"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "1af123e0a3d1380756cb5a8783f2f3f2"
server_key_iv: "5a378f8f9dbd9740c529bef1b8b9f7c3"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.746795 4589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:42.747661 4604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.747887 4589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:42.748196 4589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "9213ab98636c426bb4911aa50e9732cd"
format_stamp: "Formatted at 2026-09-25 10:12:42 on dist-test-slave-dzwq"
server_key: "1af123e0a3d1380756cb5a8783f2f3f2"
server_key_iv: "5a378f8f9dbd9740c529bef1b8b9f7c3"
server_key_version: "encryptionkey@0"
I20260925 10:12:42.748325 4589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:42.769879 4589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:42.772306 4589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:42.772492 4589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:42.772758 4589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:42.773135 4589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:42.773206 4589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.773257 4589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:42.773285 4589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:42.780061 4589 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:32911
I20260925 10:12:42.780134 4718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:32911 every 8 connection(s)
I20260925 10:12:42.780514 4589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:42.785085 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4589
I20260925 10:12:42.785194 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:42.785348 3683 external_mini_cluster.cc:1428] Setting key 30db09ca89fb122d7ce170ada9d8d9d8
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:42.791203 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.781338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:47769 (local address 127.3.152.254:40271)
0925 10:12:42.781450 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.781452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.781655 (+ 203us) server_negotiation.cc:407] Connection header received
0925 10:12:42.782864 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.782865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.782878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.782951 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.783807 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.784386 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.785724 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.786033 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.786116 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.788730 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.788731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.788731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.788739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.789936 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.790307 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.790307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.790308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.790360 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.790585 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.790586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.790587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.790718 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.790786 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.791042 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.791109 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:12:42.791347 4721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.781304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:40271 (local address 127.3.152.195:47769)
0925 10:12:42.781520 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.781560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.782727 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.783032 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.783037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.783473 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:42.783707 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.783712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.784486 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.784486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.785607 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.785610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.786136 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.786137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.786282 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.786885 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:42.786894 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:42.788639 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:42.790004 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.790010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.790012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.790230 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.790411 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:42.790411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:42.790411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:42.790503 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:42.790783 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:42.790788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:42.790906 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.791008 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.791130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260925 10:12:42.791929 4719 heartbeater.cc:347] Connected to a master server at 127.3.152.254:40271
I20260925 10:12:42.792064 4719 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:42.792220 4719 heartbeater.cc:511] Master 127.3.152.254:40271 requested a full tablet report, sending...
I20260925 10:12:42.792596 4256 ts_manager.cc:194] Registered new tserver with Master: 9213ab98636c426bb4911aa50e9732cd (127.3.152.195:32911)
I20260925 10:12:42.792943 4256 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:47769
I20260925 10:12:42.797407 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:42.805351 4301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.798907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38084 (local address 127.3.152.254:40271)
0925 10:12:42.798987 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.798990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.799086 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:12:42.799172 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.799173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.799183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.799270 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.799809 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.800380 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.801105 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.801318 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.801403 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.802682 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.802684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.802684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.802692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.804066 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.804433 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.804434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.804435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.804498 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.804699 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.804700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.804700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.804865 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.805008 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.805144 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.805230 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:12:42.807618 4256 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:42.809267 4256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:42.815948 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.3.152.193:36281)
0925 10:12:42.812084 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.812089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.812100 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:42.812143 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.812145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.812158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.812228 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:42.812446 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.813342 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.814840 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.815495 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.815592 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.815658 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.815830 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.815873 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260925 10:12:42.815932 4734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:33001 (local address 127.0.0.1:50294)
0925 10:12:42.811917 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.811926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.811986 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.812366 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.812367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.812370 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:42.812428 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.812430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.813757 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.813758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.814814 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.814816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.815629 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.815630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.815696 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.815707 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.815750 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.815812 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260925 10:12:42.816159 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.3.152.194:33001)
0925 10:12:42.812169 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.812173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.812181 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:42.812243 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.812245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.812262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.812345 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:42.812500 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.813601 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.814879 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.815527 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.815910 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.816007 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.816035 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.816077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":82,"threads_started":1}
I20260925 10:12:42.816215 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:36281 (local address 127.0.0.1:42578)
0925 10:12:42.812035 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.812041 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.812065 (+ 24us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.812287 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.812290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.812293 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:42.812371 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.812373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.813450 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.813451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.814744 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.814746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.815590 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.815591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.815686 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.815702 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.815809 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.815857 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":60,"threads_started":1}
I20260925 10:12:42.816572 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:32911 (local address 127.0.0.1:40182)
0925 10:12:42.811712 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.811754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.811819 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.812802 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.812803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.812804 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:42.812852 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.812853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.814016 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.814017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.815108 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.815111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.816132 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.816133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.816196 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.816201 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.816465 (+ 264us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.816509 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260925 10:12:42.816802 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.811628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.3.152.195:32911)
0925 10:12:42.812618 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.812622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.812631 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:42.812666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.812668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.812683 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.812805 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:42.812913 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.813909 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.815200 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.815935 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.816057 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.816107 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.816678 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.816721 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":106,"threads_started":1}
I20260925 10:12:42.816876 4516 tablet_service.cc:1467] Processing CreateTablet for tablet 0149e74f57054c7b9995972edd140e2c (DEFAULT_TABLE table=test-table [id=3850235f132b4872999275a9ffa039d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:42.817245 4516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149e74f57054c7b9995972edd140e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:42.817196 4381 tablet_service.cc:1467] Processing CreateTablet for tablet 0149e74f57054c7b9995972edd140e2c (DEFAULT_TABLE table=test-table [id=3850235f132b4872999275a9ffa039d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:42.817505 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149e74f57054c7b9995972edd140e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:42.817552 4652 tablet_service.cc:1467] Processing CreateTablet for tablet 0149e74f57054c7b9995972edd140e2c (DEFAULT_TABLE table=test-table [id=3850235f132b4872999275a9ffa039d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:42.817854 4652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0149e74f57054c7b9995972edd140e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:42.820376 4736 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: Bootstrap starting.
I20260925 10:12:42.821038 4736 tablet_bootstrap.cc:654] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:42.821075 4738 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: Bootstrap starting.
I20260925 10:12:42.821180 4737 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Bootstrap starting.
I20260925 10:12:42.821619 4736 log.cc:824] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:42.821759 4737 tablet_bootstrap.cc:654] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:42.821753 4738 tablet_bootstrap.cc:654] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:42.822327 4737 log.cc:824] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:42.822361 4738 log.cc:824] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:42.823150 4736 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: No bootstrap required, opened a new log
I20260925 10:12:42.823161 4738 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: No bootstrap required, opened a new log
I20260925 10:12:42.823227 4736 ts_tablet_manager.cc:1397] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:12:42.823227 4738 ts_tablet_manager.cc:1397] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:42.824280 4737 tablet_bootstrap.cc:492] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: No bootstrap required, opened a new log
I20260925 10:12:42.824363 4737 ts_tablet_manager.cc:1397] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:12:42.825273 4738 raft_consensus.cc:348] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.825246 4736 raft_consensus.cc:348] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.825449 4738 raft_consensus.cc:374] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:42.825449 4736 raft_consensus.cc:374] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:42.825486 4736 raft_consensus.cc:729] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b78dc2d9eed4e179eb1f23a0fefae94, State: Initialized, Role: FOLLOWER
I20260925 10:12:42.825486 4738 raft_consensus.cc:729] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9213ab98636c426bb4911aa50e9732cd, State: Initialized, Role: FOLLOWER
I20260925 10:12:42.825608 4736 consensus_queue.cc:260] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.825637 4738 consensus_queue.cc:260] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.825928 4736 ts_tablet_manager.cc:1428] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:42.825928 4738 ts_tablet_manager.cc:1428] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:42.826079 4448 heartbeater.cc:503] Master 127.3.152.254:40271 was elected leader, sending a full tablet report...
I20260925 10:12:42.826123 4719 heartbeater.cc:503] Master 127.3.152.254:40271 was elected leader, sending a full tablet report...
I20260925 10:12:42.826094 4737 raft_consensus.cc:348] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.826259 4737 raft_consensus.cc:374] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:42.826289 4737 raft_consensus.cc:729] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248bf3288d094518833c9153a4a96744, State: Initialized, Role: FOLLOWER
I20260925 10:12:42.826433 4737 consensus_queue.cc:260] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.826743 4737 ts_tablet_manager.cc:1428] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:12:42.826753 4583 heartbeater.cc:503] Master 127.3.152.254:40271 was elected leader, sending a full tablet report...
I20260925 10:12:42.835848 4744 raft_consensus.cc:482] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:42.836023 4744 raft_consensus.cc:504] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.836491 4744 leader_election.cc:288] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9213ab98636c426bb4911aa50e9732cd (127.3.152.195:32911), 9b78dc2d9eed4e179eb1f23a0fefae94 (127.3.152.193:36281)
I20260925 10:12:42.839398 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.836652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:32911 (local address 127.3.152.194:34655)
0925 10:12:42.836710 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.836723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.836768 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.836977 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.836978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.836980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:42.837047 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.837054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.837790 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.837792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.838543 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.838545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.839123 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.839124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.839211 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.839214 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.839267 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.839309 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:12:42.839439 4732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.836720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:34655 (local address 127.3.152.195:32911)
0925 10:12:42.836841 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.836843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.836853 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:42.836883 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.836883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.836895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.836955 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:42.837122 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.837700 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.838638 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.839027 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.839149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.839160 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.839306 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.839355 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 10:12:42.839545 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.836873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:38811 (local address 127.3.152.193:36281)
0925 10:12:42.836947 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.836949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.837038 (+ 89us) server_negotiation.cc:407] Connection header received
0925 10:12:42.837093 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.837094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.837107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.837167 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:42.837360 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.837814 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.838743 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.839159 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.839248 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.839256 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.839421 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.839470 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:12:42.839745 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.836777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:36281 (local address 127.3.152.194:38811)
0925 10:12:42.836972 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:42.836980 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:42.837038 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:42.837179 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:42.837179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:42.837181 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:42.837253 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.837255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.837896 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.837897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.838647 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:42.838648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.839257 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.839258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:42.839335 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.839339 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:42.839620 (+ 281us) client_negotiation.cc:240] Negotiation successful
0925 10:12:42.839676 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260925 10:12:42.839785 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149e74f57054c7b9995972edd140e2c" candidate_uuid: "248bf3288d094518833c9153a4a96744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9213ab98636c426bb4911aa50e9732cd" is_pre_election: true
I20260925 10:12:42.839931 4672 raft_consensus.cc:2454] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248bf3288d094518833c9153a4a96744 in term 0.
I20260925 10:12:42.840209 4469 leader_election.cc:302] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248bf3288d094518833c9153a4a96744, 9213ab98636c426bb4911aa50e9732cd; no voters:
I20260925 10:12:42.840324 4744 raft_consensus.cc:2781] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:42.840379 4744 raft_consensus.cc:482] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:42.840329 4402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149e74f57054c7b9995972edd140e2c" candidate_uuid: "248bf3288d094518833c9153a4a96744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" is_pre_election: true
I20260925 10:12:42.840401 4744 raft_consensus.cc:3037] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:42.840473 4402 raft_consensus.cc:2454] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248bf3288d094518833c9153a4a96744 in term 0.
I20260925 10:12:42.841190 4744 raft_consensus.cc:504] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.841326 4744 leader_election.cc:288] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [CANDIDATE]: Term 1 election: Requested vote from peers 9213ab98636c426bb4911aa50e9732cd (127.3.152.195:32911), 9b78dc2d9eed4e179eb1f23a0fefae94 (127.3.152.193:36281)
I20260925 10:12:42.841518 4402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149e74f57054c7b9995972edd140e2c" candidate_uuid: "248bf3288d094518833c9153a4a96744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94"
I20260925 10:12:42.841528 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0149e74f57054c7b9995972edd140e2c" candidate_uuid: "248bf3288d094518833c9153a4a96744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9213ab98636c426bb4911aa50e9732cd"
I20260925 10:12:42.841594 4672 raft_consensus.cc:3037] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:42.841616 4402 raft_consensus.cc:3037] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:42.842289 4672 raft_consensus.cc:2454] T 0149e74f57054c7b9995972edd140e2c P 9213ab98636c426bb4911aa50e9732cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248bf3288d094518833c9153a4a96744 in term 1.
I20260925 10:12:42.842368 4402 raft_consensus.cc:2454] T 0149e74f57054c7b9995972edd140e2c P 9b78dc2d9eed4e179eb1f23a0fefae94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248bf3288d094518833c9153a4a96744 in term 1.
I20260925 10:12:42.842446 4469 leader_election.cc:302] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248bf3288d094518833c9153a4a96744, 9213ab98636c426bb4911aa50e9732cd; no voters:
I20260925 10:12:42.842543 4744 raft_consensus.cc:2781] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:42.842715 4744 raft_consensus.cc:686] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [term 1 LEADER]: Becoming Leader. State: Replica: 248bf3288d094518833c9153a4a96744, State: Running, Role: LEADER
I20260925 10:12:42.842845 4744 consensus_queue.cc:237] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } }
I20260925 10:12:42.843685 4255 catalog_manager.cc:5495] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744 reported cstate change: term changed from 0 to 1, leader changed from <none> to 248bf3288d094518833c9153a4a96744 (127.3.152.194). New cstate: current_term: 1 leader_uuid: "248bf3288d094518833c9153a4a96744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9213ab98636c426bb4911aa50e9732cd" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b78dc2d9eed4e179eb1f23a0fefae94" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248bf3288d094518833c9153a4a96744" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 33001 } health_report { overall_health: HEALTHY } } }
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:42.855792 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.848364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.3.152.193:36281)
0925 10:12:42.848424 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.848426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.848496 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:12:42.848551 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.848552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.848562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.848673 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.849241 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.849773 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.850486 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.850743 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.850803 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.853260 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.853269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.853270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.853283 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.854749 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.855073 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.855077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.855077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.855152 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.855323 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.855323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.855324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.855463 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.855566 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.855649 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.855720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:12:42.856247 4584 tablet.cc:2369] T 0149e74f57054c7b9995972edd140e2c P 248bf3288d094518833c9153a4a96744: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:42.879660 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.872425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.3.152.193:36281)
0925 10:12:42.872501 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.872503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.872595 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:12:42.872742 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.872743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.872753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.872838 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.873462 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.873951 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.874703 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.874901 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.875006 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.877261 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.877263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.877264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.877271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.878429 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.878791 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.878792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.878792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.878853 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.879058 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.879059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.879059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.879162 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.879260 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.879451 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.879535 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:42.902475 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:42.895636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.3.152.193:36281)
0925 10:12:42.895755 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:42.895756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:42.895805 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:12:42.895860 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:42.895860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:42.895869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:42.895931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:42.896540 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.897059 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.897875 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:42.898056 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:42.898111 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:42.900433 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:42.900435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:42.900435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:42.900442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:42.901609 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.901913 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.901914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.901914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.901964 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:42.902113 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:42.902114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:42.902114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:42.902196 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:42.902275 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:42.902349 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:12:42.902407 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260925 10:12:42.902998 4361 tablet_service.cc:2908] Rejecting scan request for tablet 0149e74f57054c7b9995972edd140e2c: Uninitialized: clean time has not yet been initialized
I20260925 10:12:42.903834 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4318
I20260925 10:12:42.908598 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4453
I20260925 10:12:42.913249 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4589
I20260925 10:12:42.917804 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4223
2026-09-25T10:12:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1955 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4760](info): set up 2 sockets
Sep 25 10:12:42 dist-test-slave-dzwq krb5kdc[4760](info): commencing operation
krb5kdc: starting...
W20260925 10:12:43.968215 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:44.091840 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36269
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:36269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:44.175056 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:44.175278 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:44.175302 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:44.176569 4773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:44.176620 4773 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:12:44.176633 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:44.176646 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:44.176659 4773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:44.176717 4773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:44.178278 4773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:36269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:44.178560 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:44.178830 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:12:44.184167 4773 server_base.cc:1034] running on GCE node
W20260925 10:12:44.184132 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.184134 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.184144 4778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.184713 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:44.185045 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:44.186229 4773 hybrid_clock.cc:648] HybridClock initialized: now 1790331164186199 us; error 57 us; skew 500 ppm
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:44.188939 4773 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:44.189476 4773 webserver.cc:460] Webserver started at http://127.3.152.254:39351/ using document root <none> and password file <none>
I20260925 10:12:44.189749 4773 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:44.189833 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:44.189988 4773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:44.191089 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0a92ceeea9394c97ae97f925ff2b3011"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "71386bd5249569e06c4220853734b745"
server_key_iv: "eeef75bdd89d2848858ea3f830a37bb4"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.191469 4773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0a92ceeea9394c97ae97f925ff2b3011"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "71386bd5249569e06c4220853734b745"
server_key_iv: "eeef75bdd89d2848858ea3f830a37bb4"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.192963 4773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:44.193811 4788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.194000 4773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:44.194121 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0a92ceeea9394c97ae97f925ff2b3011"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "71386bd5249569e06c4220853734b745"
server_key_iv: "eeef75bdd89d2848858ea3f830a37bb4"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.194206 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:44.205001 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:44.206997 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:44.207191 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:44.212599 4773 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:36269
I20260925 10:12:44.212636 4851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:36269 every 8 connection(s)
I20260925 10:12:44.213050 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:44.213778 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.216295 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Bootstrap starting.
I20260925 10:12:44.216888 4852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.217077 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4773
I20260925 10:12:44.217172 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:44.217262 4852 log.cc:824] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:44.217360 3683 external_mini_cluster.cc:1428] Setting key 5b1241ff0ebf43ca46680aaf1d1e9d6f
I20260925 10:12:44.218107 4852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: No bootstrap required, opened a new log
I20260925 10:12:44.220521 4852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } }
I20260925 10:12:44.220686 4852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.220710 4852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a92ceeea9394c97ae97f925ff2b3011, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.220875 4852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } }
I20260925 10:12:44.220971 4852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:44.221005 4852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:44.221057 4852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:44.221834 4852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } }
I20260925 10:12:44.221935 4852 leader_election.cc:302] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a92ceeea9394c97ae97f925ff2b3011; no voters:
I20260925 10:12:44.222162 4852 leader_election.cc:288] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:44.222208 4857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:44.222353 4857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [term 1 LEADER]: Becoming Leader. State: Replica: 0a92ceeea9394c97ae97f925ff2b3011, State: Running, Role: LEADER
I20260925 10:12:44.222450 4852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:44.222441 4857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } }
W20260925 10:12:44.224619 4871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:12:44.224673 4871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:12:44.223934 4858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a92ceeea9394c97ae97f925ff2b3011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } } }
I20260925 10:12:44.224146 4859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a92ceeea9394c97ae97f925ff2b3011. Latest consensus state: current_term: 1 leader_uuid: "0a92ceeea9394c97ae97f925ff2b3011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a92ceeea9394c97ae97f925ff2b3011" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36269 } } }
I20260925 10:12:44.225131 4859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:44.225277 4858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:44.225370 4874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:44.225574 4874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:44.227567 4874 catalog_manager.cc:1347] Generated new cluster ID: 5dd3ef95ac154d51809702c6fac38ace
I20260925 10:12:44.227622 4874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:44.231683 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.217929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.3.152.254:36269)
0925 10:12:44.218181 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.218193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.218237 (+ 44us) server_negotiation.cc:407] Connection header received
0925 10:12:44.220516 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.220525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.220585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.220790 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.221613 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.223073 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.224328 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.224908 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.224995 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.227845 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.227848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.227849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.227860 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.229940 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.230377 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.230380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.230380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.230499 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.230726 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.230727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.230727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.231039 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.231201 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.231324 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.231462 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260925 10:12:44.234345 4874 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:44.235100 4874 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:44.238258 4874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Generated new TSK 0
I20260925 10:12:44.238480 4874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:44.266947 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:36269
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:44.352685 4878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:44.352905 4878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:44.352932 4878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:44.354206 4878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:44.354251 4878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:44.354272 4878 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:44.354334 4878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:44.355829 4878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:44.356122 4878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:44.356401 4878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:44.361784 4886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.361899 4883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.361969 4878 server_base.cc:1034] running on GCE node
W20260925 10:12:44.361784 4884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.362263 4878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:44.362514 4878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:44.363739 4878 hybrid_clock.cc:648] HybridClock initialized: now 1790331164363726 us; error 45 us; skew 500 ppm
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:44.366405 4878 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:44.366998 4878 webserver.cc:460] Webserver started at http://127.3.152.193:33453/ using document root <none> and password file <none>
I20260925 10:12:44.367259 4878 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:44.367338 4878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:44.367507 4878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:44.368774 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "4439a87277f14db4a0d5286b2af05a86"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "1d47669723754c387b2d603b8e092520"
server_key_iv: "6dc0f2a88f372295f305915dcd29b95b"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.369184 4878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "4439a87277f14db4a0d5286b2af05a86"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "1d47669723754c387b2d603b8e092520"
server_key_iv: "6dc0f2a88f372295f305915dcd29b95b"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.370693 4878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:44.371474 4893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.371681 4878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:44.371790 4878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "4439a87277f14db4a0d5286b2af05a86"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "1d47669723754c387b2d603b8e092520"
server_key_iv: "6dc0f2a88f372295f305915dcd29b95b"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.371883 4878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:44.383767 4878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:44.385912 4878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:44.386039 4878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:44.386600 4878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:44.386663 4878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.386696 4878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:44.386716 4878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:44.393863 4878 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35717
I20260925 10:12:44.393951 5009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35717 every 8 connection(s)
I20260925 10:12:44.394305 4878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:44.397423 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.387123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:44083 (local address 127.3.152.254:36269)
0925 10:12:44.387186 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.387188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.387413 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:12:44.387985 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.387987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.388005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.388070 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.388794 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.389313 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.390463 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.390813 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.390903 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.394080 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.394082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.394082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.394095 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.395803 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.396292 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.396294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.396294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.396357 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.396656 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.396657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.396658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.396789 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.396868 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.397168 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.397328 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:12:44.397570 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.387098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.193:44083)
0925 10:12:44.387280 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.387322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.387857 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.388152 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.388174 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.388459 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.388674 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.388681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.389413 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.389414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.390362 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.390364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.390915 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.390916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.391072 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.391641 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.391650 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.393970 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.395918 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.395926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.395928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.396199 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.396429 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.396430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.396430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.396588 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.396859 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.396866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.397016 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.397134 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.397332 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260925 10:12:44.398150 5010 heartbeater.cc:347] Connected to a master server at 127.3.152.254:36269
I20260925 10:12:44.398248 5010 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:44.398459 5010 heartbeater.cc:511] Master 127.3.152.254:36269 requested a full tablet report, sending...
I20260925 10:12:44.398957 4805 ts_manager.cc:194] Registered new tserver with Master: 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717)
I20260925 10:12:44.399731 4805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:44083
I20260925 10:12:44.402380 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 4878
I20260925 10:12:44.402634 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:44.402848 3683 external_mini_cluster.cc:1428] Setting key 376d4cbd095f661251074a11a4230f0a
I20260925 10:12:44.405731 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.399361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:52679 (local address 127.3.152.254:36269)
0925 10:12:44.399407 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.399409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.399458 (+ 49us) server_negotiation.cc:407] Connection header received
0925 10:12:44.399523 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.399524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.399532 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.399585 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.399954 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.400794 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.401676 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.401883 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.401947 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.403020 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.403022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.403022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.403030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.404547 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.404929 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.404931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.404931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.404994 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.405197 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.405198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.405198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.405313 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.405386 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.405554 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.405637 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:44.405736 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.399283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.193:52679)
0925 10:12:44.399400 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.399410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.399470 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.399588 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.399589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.399747 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.399831 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.399834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.400927 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.400928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.401575 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.401578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.401956 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.401957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.402013 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.402502 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.402507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.402944 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.404676 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.404677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.404677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.404856 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.405059 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.405060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.405060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.405120 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.405391 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.405392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.405440 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.405528 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.405638 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:44.430954 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:36269
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:44.515151 5019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:44.515370 5019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:44.515395 5019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:44.516695 5019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:44.516743 5019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:44.516765 5019 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:44.516782 5019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:44.518302 5019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:44.518558 5019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:44.518803 5019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:44.524188 5024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.524188 5019 server_base.cc:1034] running on GCE node
W20260925 10:12:44.524257 5027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.524379 5025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.524655 5019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:44.524904 5019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:44.526085 5019 hybrid_clock.cc:648] HybridClock initialized: now 1790331164526067 us; error 35 us; skew 500 ppm
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:44.529394 5019 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:44.529938 5019 webserver.cc:460] Webserver started at http://127.3.152.194:36953/ using document root <none> and password file <none>
I20260925 10:12:44.530234 5019 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:44.530314 5019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:44.530481 5019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:44.531639 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "7ac4363eb0f24649bd2015fa231d68fd"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "8b1bdf4443b9e299f2df9c04963e1ad8"
server_key_iv: "73ebc14c3cb2c9eff2344ef08914f160"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.532110 5019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac4363eb0f24649bd2015fa231d68fd"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "8b1bdf4443b9e299f2df9c04963e1ad8"
server_key_iv: "73ebc14c3cb2c9eff2344ef08914f160"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.533721 5019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:44.534708 5034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.534960 5019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:44.535100 5019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "7ac4363eb0f24649bd2015fa231d68fd"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "8b1bdf4443b9e299f2df9c04963e1ad8"
server_key_iv: "73ebc14c3cb2c9eff2344ef08914f160"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.535212 5019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:44.557327 5019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:44.560181 5019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:44.560377 5019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:44.561002 5019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:44.561092 5019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.561142 5019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:44.561177 5019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:44.569525 5019 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35451
I20260925 10:12:44.569670 5149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35451 every 8 connection(s)
I20260925 10:12:44.570020 5019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:44.572983 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.561621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:49001 (local address 127.3.152.254:36269)
0925 10:12:44.561744 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.561747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.561961 (+ 214us) server_negotiation.cc:407] Connection header received
0925 10:12:44.562769 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.562770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.562791 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.562881 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.563923 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.564839 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.566586 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.566830 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.566915 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.569890 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.569892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.569892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.569900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.571402 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.571865 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.571866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.571866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.571924 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.572290 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.572291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.572291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.572407 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.572463 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.572836 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.572914 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 10:12:44.573076 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.561543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.194:49001)
0925 10:12:44.561788 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.561841 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.562625 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.562985 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.562989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.563463 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.563803 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.563810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.564947 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.564949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.566478 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.566483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.566956 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.566957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.567142 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.567738 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.567750 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.569770 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.571507 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.571514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.571516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.571787 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.572093 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.572094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.572094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.572213 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.572506 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.572512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.572648 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.572761 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.572905 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260925 10:12:44.573748 5150 heartbeater.cc:347] Connected to a master server at 127.3.152.254:36269
I20260925 10:12:44.573832 5150 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:44.574041 5150 heartbeater.cc:511] Master 127.3.152.254:36269 requested a full tablet report, sending...
I20260925 10:12:44.574458 4804 ts_manager.cc:194] Registered new tserver with Master: 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194:35451)
I20260925 10:12:44.574887 4804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:49001
I20260925 10:12:44.576351 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5019
I20260925 10:12:44.576432 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:44.576617 3683 external_mini_cluster.cc:1428] Setting key a131f56e6993c8b3d8f5b62ebc1430f2
I20260925 10:12:44.581538 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.574896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.194:47373)
0925 10:12:44.575083 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.575094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.575176 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.575303 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.575303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.575485 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.575543 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.575544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.576328 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.576328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.577053 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.577055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.577480 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.577481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.577558 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.578059 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.578071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.578362 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.580264 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.580265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.580265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.580518 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.580750 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.580751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.580751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.580810 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.581218 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.581220 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.581283 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.581372 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.581464 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":132,"threads_started":1}
I20260925 10:12:44.581624 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.574954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:47373 (local address 127.3.152.254:36269)
0925 10:12:44.575005 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.575007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.575161 (+ 154us) server_negotiation.cc:407] Connection header received
0925 10:12:44.575235 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.575236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.575245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.575289 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.575619 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.576195 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.577141 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.577385 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.577478 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.578472 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.578473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.578474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.578481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.580167 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.580614 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.580615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.580615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.580680 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.580873 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.580874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.580875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.581095 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.581174 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.581425 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.581512 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:44.607007 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:36269
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:44.689769 5159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:44.690040 5159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:44.690083 5159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:44.691298 5159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:44.691375 5159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:44.691411 5159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:44.691448 5159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:44.692996 5159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:36269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:44.693298 5159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:44.693544 5159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:44.698956 5165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.698966 5167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:44.699043 5164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:44.699558 5159 server_base.cc:1034] running on GCE node
I20260925 10:12:44.699827 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:44.700137 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:44.701326 5159 hybrid_clock.cc:648] HybridClock initialized: now 1790331164701315 us; error 24 us; skew 500 ppm
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:44.703939 5159 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:44.704499 5159 webserver.cc:460] Webserver started at http://127.3.152.195:45019/ using document root <none> and password file <none>
I20260925 10:12:44.704774 5159 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:44.704823 5159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:44.704962 5159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:44.706049 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "af03440c98e04169a5b14200161663df"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "c165e4935f8568aa4dc4713ba3ca8e5e"
server_key_iv: "6935416532adc0b134520c7632c10ed5"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.706420 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "af03440c98e04169a5b14200161663df"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "c165e4935f8568aa4dc4713ba3ca8e5e"
server_key_iv: "6935416532adc0b134520c7632c10ed5"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.707839 5159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 10:12:44.708796 5174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.709008 5159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:44.709123 5159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "af03440c98e04169a5b14200161663df"
format_stamp: "Formatted at 2026-09-25 10:12:44 on dist-test-slave-dzwq"
server_key: "c165e4935f8568aa4dc4713ba3ca8e5e"
server_key_iv: "6935416532adc0b134520c7632c10ed5"
server_key_version: "encryptionkey@0"
I20260925 10:12:44.709188 5159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:44.750864 5159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:44.753749 5159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:44.753887 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:44.754433 5159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:44.754479 5159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.754506 5159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:44.754566 5159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.762164 5159 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:37707
I20260925 10:12:44.762220 5290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:37707 every 8 connection(s)
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:44.762578 5159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:44.762868 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5159
I20260925 10:12:44.762986 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:44.763188 3683 external_mini_cluster.cc:1428] Setting key eb4fceb975af428067ee5b1189e0a474
I20260925 10:12:44.766314 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.755005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39585 (local address 127.3.152.254:36269)
0925 10:12:44.755083 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.755086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.755376 (+ 290us) server_negotiation.cc:407] Connection header received
0925 10:12:44.756029 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.756030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.756050 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.756100 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.756985 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.757810 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.759723 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.760056 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.760119 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.763151 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.763154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.763154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.763163 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.764923 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.765319 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.765320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.765320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.765376 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.765700 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.765701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.765701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.765841 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.765907 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.766176 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.766247 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:12:44.766481 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.754980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.195:39585)
0925 10:12:44.755263 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.755293 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.755808 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.756192 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.756199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.756597 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.756898 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.756904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.757932 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.757933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.759608 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.759612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.760148 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.760150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.760362 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.761098 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.761110 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.763043 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.765002 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.765008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.765009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.765228 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.765442 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.765442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.765443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.765552 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.765932 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.765938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.766059 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.766142 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.766278 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260925 10:12:44.766978 5291 heartbeater.cc:347] Connected to a master server at 127.3.152.254:36269
I20260925 10:12:44.767109 5291 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:44.767299 5291 heartbeater.cc:511] Master 127.3.152.254:36269 requested a full tablet report, sending...
I20260925 10:12:44.767758 4804 ts_manager.cc:194] Registered new tserver with Master: af03440c98e04169a5b14200161663df (127.3.152.195:37707)
I20260925 10:12:44.768294 4804 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:39585
I20260925 10:12:44.773919 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.768045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.3.152.195:45269)
0925 10:12:44.768228 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.768239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.768313 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.768466 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.768467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.768684 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:44.768741 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.768743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.769479 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.769480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.769970 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.769972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.770304 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.770304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.770357 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.770839 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:44.770846 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:44.771120 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:44.772894 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.772895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.772896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.773074 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.773308 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:44.773308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:44.773308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:44.773357 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:44.773580 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:44.773581 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:44.773616 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.773719 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.773848 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":136,"threads_started":1}
I20260925 10:12:44.773976 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.768083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:45269 (local address 127.3.152.254:36269)
0925 10:12:44.768129 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.768131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.768297 (+ 166us) server_negotiation.cc:407] Connection header received
0925 10:12:44.768388 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.768389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.768399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.768452 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.768821 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.769395 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.770044 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.770235 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.770290 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.771209 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.771210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.771210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.771215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.772806 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.773161 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.773163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.773163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.773238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.773414 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.773414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.773415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.773524 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.773580 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.773792 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.773870 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:44.775414 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:12:44 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:44.799347 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.792287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.3.152.254:36269)
0925 10:12:44.792351 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.792353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.792363 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:44.792399 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.792400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.792409 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.792464 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:44.793017 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.793510 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.794230 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.794407 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.794458 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.796839 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:44.796840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:44.796841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:44.796847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:44.798293 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.798738 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.798739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.798739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.798791 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:44.798955 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:44.798955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:44.798955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:44.799054 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:44.799123 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.799216 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.799279 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:12:44.801107 4804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:44.802718 4804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:44.809643 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35451 (local address 127.0.0.1:51224)
0925 10:12:44.805384 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.805426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.805498 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.805810 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.805813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.805815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:44.805892 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.805895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.807475 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.807476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.808560 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.808562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809270 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.809367 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809390 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.809477 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.809546 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":26,"thread_start_us":55,"threads_started":1}
I20260925 10:12:44.809674 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.3.152.195:37707)
0925 10:12:44.805833 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.805837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.805848 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:44.805892 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.805894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.805911 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.805988 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:44.806195 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.806937 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.808416 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809178 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809263 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809324 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.809532 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.809574 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":75,"threads_started":1}
I20260925 10:12:44.809737 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.3.152.193:35717)
0925 10:12:44.806003 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.806007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.806017 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:44.806055 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.806057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.806073 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.806153 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:44.806273 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.807366 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.808677 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809387 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809460 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809513 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.809588 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.809625 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":79,"threads_started":1}
I20260925 10:12:44.810068 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.3.152.194:35451)
0925 10:12:44.805620 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.805625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.805636 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:44.805680 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.805683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.805700 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.805796 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:44.806092 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.807360 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.808647 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809158 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809267 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.809928 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.809971 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260925 10:12:44.809710 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35717 (local address 127.0.0.1:38950)
0925 10:12:44.805892 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.805902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.805949 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.806159 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.806160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.806162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:44.806195 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.806196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.807467 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.807468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.808579 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.808583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809468 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.809524 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809527 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.809578 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.809608 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":31,"thread_start_us":127,"threads_started":1}
I20260925 10:12:44.809903 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.805192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37707 (local address 127.0.0.1:40216)
0925 10:12:44.805384 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.805426 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.805498 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.806050 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.806051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.806052 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:44.806107 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.806109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.807036 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.807037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.808326 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.808329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.809360 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.809361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.809432 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.809437 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.809486 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.809831 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":288,"thread_start_us":122,"threads_started":1}
I20260925 10:12:44.811028 5084 tablet_service.cc:1467] Processing CreateTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca (DEFAULT_TABLE table=test-table [id=ce103278d22b414eb66c84285853ab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.811013 5225 tablet_service.cc:1467] Processing CreateTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca (DEFAULT_TABLE table=test-table [id=ce103278d22b414eb66c84285853ab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.811407 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aab234abbca44578fd95c6ccbebd7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.811407 5225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aab234abbca44578fd95c6ccbebd7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.814898 5315 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Bootstrap starting.
I20260925 10:12:44.815337 4943 tablet_service.cc:1467] Processing CreateTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca (DEFAULT_TABLE table=test-table [id=ce103278d22b414eb66c84285853ab1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.815694 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aab234abbca44578fd95c6ccbebd7ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.815924 5315 tablet_bootstrap.cc:654] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.816124 5316 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Bootstrap starting.
I20260925 10:12:44.816305 5315 log.cc:824] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:44.816751 5316 tablet_bootstrap.cc:654] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.817068 5316 log.cc:824] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:44.817176 5315 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: No bootstrap required, opened a new log
I20260925 10:12:44.817241 5315 ts_tablet_manager.cc:1397] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:44.817876 5316 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: No bootstrap required, opened a new log
I20260925 10:12:44.817951 5316 ts_tablet_manager.cc:1397] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:44.818800 5315 raft_consensus.cc:348] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.818954 5315 raft_consensus.cc:374] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.818997 5315 raft_consensus.cc:729] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af03440c98e04169a5b14200161663df, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.819128 5315 consensus_queue.cc:260] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.819418 5315 ts_tablet_manager.cc:1428] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:44.819552 5291 heartbeater.cc:503] Master 127.3.152.254:36269 was elected leader, sending a full tablet report...
I20260925 10:12:44.819551 5319 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Bootstrap starting.
I20260925 10:12:44.820179 5319 tablet_bootstrap.cc:654] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.820094 5316 raft_consensus.cc:348] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.820255 5316 raft_consensus.cc:374] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.820300 5316 raft_consensus.cc:729] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac4363eb0f24649bd2015fa231d68fd, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.820411 5316 consensus_queue.cc:260] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.820523 5319 log.cc:824] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:44.820685 5316 ts_tablet_manager.cc:1428] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:12:44.820814 5150 heartbeater.cc:503] Master 127.3.152.254:36269 was elected leader, sending a full tablet report...
I20260925 10:12:44.821352 5319 tablet_bootstrap.cc:492] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: No bootstrap required, opened a new log
I20260925 10:12:44.821425 5319 ts_tablet_manager.cc:1397] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:44.823015 5319 raft_consensus.cc:348] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.823158 5319 raft_consensus.cc:374] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.823203 5319 raft_consensus.cc:729] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439a87277f14db4a0d5286b2af05a86, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.823316 5319 consensus_queue.cc:260] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.823551 5319 ts_tablet_manager.cc:1428] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:44.823661 5010 heartbeater.cc:503] Master 127.3.152.254:36269 was elected leader, sending a full tablet report...
I20260925 10:12:44.825376 5323 raft_consensus.cc:482] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:44.825439 5323 raft_consensus.cc:504] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.825742 5323 leader_election.cc:288] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af03440c98e04169a5b14200161663df (127.3.152.195:37707), 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194:35451)
I20260925 10:12:44.828652 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.826008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:49453 (local address 127.3.152.195:37707)
0925 10:12:44.826051 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.826053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.826110 (+ 57us) server_negotiation.cc:407] Connection header received
0925 10:12:44.826200 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.826201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.826210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.826257 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:44.826411 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.826888 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.827892 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.828324 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.828418 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.828428 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.828572 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.828606 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:12:44.828678 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.825898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37707 (local address 127.3.152.193:49453)
0925 10:12:44.826043 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.826054 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.826124 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.826274 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.826275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.826277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:44.826354 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.826356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.826971 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.826972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.827797 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.827799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.828419 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.828420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.828497 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.828501 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.828557 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.828594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":49,"thread_start_us":80,"threads_started":1}
I20260925 10:12:44.829303 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.825889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35451 (local address 127.3.152.193:60443)
0925 10:12:44.825927 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:44.825939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:44.825979 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:44.826203 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:44.826204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:44.826205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:44.826300 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.826301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.827031 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.827032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.827795 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:44.827797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.829111 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.829112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:44.829179 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.829182 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:44.829224 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 10:12:44.829259 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":28}
I20260925 10:12:44.829375 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.826012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:60443 (local address 127.3.152.194:35451)
0925 10:12:44.826054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.826056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.826064 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:44.826125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.826127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.826136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.826193 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:44.826360 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.826957 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.828462 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.829027 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.829101 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.829111 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.829251 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.829295 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:44.829530 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aab234abbca44578fd95c6ccbebd7ca" candidate_uuid: "4439a87277f14db4a0d5286b2af05a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df" is_pre_election: true
I20260925 10:12:44.829629 5245 raft_consensus.cc:2454] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439a87277f14db4a0d5286b2af05a86 in term 0.
I20260925 10:12:44.829761 5104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aab234abbca44578fd95c6ccbebd7ca" candidate_uuid: "4439a87277f14db4a0d5286b2af05a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac4363eb0f24649bd2015fa231d68fd" is_pre_election: true
I20260925 10:12:44.829823 4896 leader_election.cc:302] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439a87277f14db4a0d5286b2af05a86, af03440c98e04169a5b14200161663df; no voters:
I20260925 10:12:44.829885 5104 raft_consensus.cc:2454] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4439a87277f14db4a0d5286b2af05a86 in term 0.
I20260925 10:12:44.829931 5323 raft_consensus.cc:2781] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:44.829978 5323 raft_consensus.cc:482] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:44.830009 5323 raft_consensus.cc:3037] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:44.830677 5323 raft_consensus.cc:504] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.830781 5323 leader_election.cc:288] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [CANDIDATE]: Term 1 election: Requested vote from peers af03440c98e04169a5b14200161663df (127.3.152.195:37707), 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194:35451)
I20260925 10:12:44.830943 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aab234abbca44578fd95c6ccbebd7ca" candidate_uuid: "4439a87277f14db4a0d5286b2af05a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df"
I20260925 10:12:44.830991 5104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aab234abbca44578fd95c6ccbebd7ca" candidate_uuid: "4439a87277f14db4a0d5286b2af05a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac4363eb0f24649bd2015fa231d68fd"
I20260925 10:12:44.831066 5245 raft_consensus.cc:3037] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:44.831089 5104 raft_consensus.cc:3037] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:44.831773 5104 raft_consensus.cc:2454] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439a87277f14db4a0d5286b2af05a86 in term 1.
I20260925 10:12:44.831784 5245 raft_consensus.cc:2454] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4439a87277f14db4a0d5286b2af05a86 in term 1.
I20260925 10:12:44.832024 4894 leader_election.cc:302] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4439a87277f14db4a0d5286b2af05a86, 7ac4363eb0f24649bd2015fa231d68fd; no voters:
I20260925 10:12:44.832156 5323 raft_consensus.cc:2781] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:44.832319 5323 raft_consensus.cc:686] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 1 LEADER]: Becoming Leader. State: Replica: 4439a87277f14db4a0d5286b2af05a86, State: Running, Role: LEADER
I20260925 10:12:44.832412 5323 consensus_queue.cc:237] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } }
I20260925 10:12:44.833171 4805 catalog_manager.cc:5495] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "4439a87277f14db4a0d5286b2af05a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } health_report { overall_health: HEALTHY } } }
I20260925 10:12:44.846449 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:44.843922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.3.152.193:35717)
0925 10:12:44.843996 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:44.843998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:44.844009 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:44.844087 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:44.844088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:44.844105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:44.844168 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:44.844296 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.844873 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.845675 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:44.845871 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:44.845943 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:44.846126 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:44.846222 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:44.846297 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:44.846347 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:12:44.846367 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:12:44.848289 5245 raft_consensus.cc:1261] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Refusing update from remote peer 4439a87277f14db4a0d5286b2af05a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:44.848294 5104 raft_consensus.cc:1261] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Refusing update from remote peer 4439a87277f14db4a0d5286b2af05a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:44.848651 5323 consensus_queue.cc:1035] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:44.848800 5325 consensus_queue.cc:1035] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:44.851263 5328 mvcc.cc:205] Tried to move back new op lower bound from 7333196451215757312 to 7333196451154198528. Current Snapshot: MvccSnapshot[applied={T|T < 7333196451215757312}]
I20260925 10:12:44.855103 4804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:44.855193 4804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:44.856042 4804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Sending DeleteTablet for 3 replicas of tablet 6aab234abbca44578fd95c6ccbebd7ca
I20260925 10:12:44.856400 5225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:44 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40216
I20260925 10:12:44.856479 5084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:44 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:51224
I20260925 10:12:44.856536 4943 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aab234abbca44578fd95c6ccbebd7ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:44 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:38950
I20260925 10:12:44.856784 5338 tablet_replica.cc:330] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: stopping tablet replica
I20260925 10:12:44.856848 5340 tablet_replica.cc:330] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: stopping tablet replica
I20260925 10:12:44.856956 5340 raft_consensus.cc:2229] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:44.856968 5338 raft_consensus.cc:2229] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:44.857061 5340 raft_consensus.cc:2258] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:44.857056 4804 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:44.857213 4804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:44.857398 5338 raft_consensus.cc:2258] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:44.857769 5340 ts_tablet_manager.cc:1905] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:44.858006 5338 ts_tablet_manager.cc:1905] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:44.858460 4943 tablet_service.cc:1467] Processing CreateTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 (DEFAULT_TABLE table=test-table [id=3d5a8676c9d0488eb642b236475a5b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.858584 5084 tablet_service.cc:1467] Processing CreateTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 (DEFAULT_TABLE table=test-table [id=3d5a8676c9d0488eb642b236475a5b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.858717 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bcb781c544e8bbcd40b9e6f67cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.860132 5225 tablet_service.cc:1467] Processing CreateTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 (DEFAULT_TABLE table=test-table [id=3d5a8676c9d0488eb642b236475a5b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:44.860509 5316 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Bootstrap starting.
I20260925 10:12:44.860621 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bcb781c544e8bbcd40b9e6f67cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.861259 5316 tablet_bootstrap.cc:654] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.862193 5319 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Bootstrap starting.
I20260925 10:12:44.862227 5340 ts_tablet_manager.cc:1918] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:44.862399 5340 log.cc:1196] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/6aab234abbca44578fd95c6ccbebd7ca
I20260925 10:12:44.862424 5338 ts_tablet_manager.cc:1918] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:44.862491 5338 log.cc:1196] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/6aab234abbca44578fd95c6ccbebd7ca
I20260925 10:12:44.862617 5319 tablet_bootstrap.cc:654] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.862787 5338 ts_tablet_manager.cc:1939] T 6aab234abbca44578fd95c6ccbebd7ca P af03440c98e04169a5b14200161663df: Deleting consensus metadata
I20260925 10:12:44.862884 5340 ts_tablet_manager.cc:1939] T 6aab234abbca44578fd95c6ccbebd7ca P 4439a87277f14db4a0d5286b2af05a86: Deleting consensus metadata
I20260925 10:12:44.862968 5225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b28bcb781c544e8bbcd40b9e6f67cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:44.863292 4791 catalog_manager.cc:4841] TS af03440c98e04169a5b14200161663df (127.3.152.195:37707): tablet 6aab234abbca44578fd95c6ccbebd7ca (table test-table [id=ce103278d22b414eb66c84285853ab1f]) successfully deleted
I20260925 10:12:44.863457 4792 catalog_manager.cc:4841] TS 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717): tablet 6aab234abbca44578fd95c6ccbebd7ca (table test-table [id=ce103278d22b414eb66c84285853ab1f]) successfully deleted
I20260925 10:12:44.863792 5316 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: No bootstrap required, opened a new log
I20260925 10:12:44.863878 5316 ts_tablet_manager.cc:1397] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:12:44.864079 5316 raft_consensus.cc:348] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.864176 5316 raft_consensus.cc:374] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.864203 5316 raft_consensus.cc:729] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac4363eb0f24649bd2015fa231d68fd, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.864254 5315 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Bootstrap starting.
I20260925 10:12:44.864255 5316 consensus_queue.cc:260] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.864343 5316 ts_tablet_manager.cc:1428] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:44.864647 5315 tablet_bootstrap.cc:654] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:44.865504 5319 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: No bootstrap required, opened a new log
I20260925 10:12:44.865581 5319 ts_tablet_manager.cc:1397] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:12:44.865739 5319 raft_consensus.cc:348] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.865830 5319 raft_consensus.cc:374] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.865862 5319 raft_consensus.cc:729] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439a87277f14db4a0d5286b2af05a86, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.865911 5319 consensus_queue.cc:260] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.866003 5319 ts_tablet_manager.cc:1428] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:44.868117 5339 tablet_replica.cc:330] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: stopping tablet replica
I20260925 10:12:44.868276 5339 raft_consensus.cc:2229] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:44.868413 5339 raft_consensus.cc:2258] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:44.868965 5339 ts_tablet_manager.cc:1905] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:44.869055 5315 tablet_bootstrap.cc:492] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: No bootstrap required, opened a new log
I20260925 10:12:44.869122 5315 ts_tablet_manager.cc:1397] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 10:12:44.869282 5315 raft_consensus.cc:348] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.869342 5315 raft_consensus.cc:374] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:44.869376 5315 raft_consensus.cc:729] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af03440c98e04169a5b14200161663df, State: Initialized, Role: FOLLOWER
I20260925 10:12:44.869427 5315 consensus_queue.cc:260] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:44.869545 5315 ts_tablet_manager.cc:1428] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:44.870895 5339 ts_tablet_manager.cc:1918] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:12:44.870976 5339 log.cc:1196] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/6aab234abbca44578fd95c6ccbebd7ca
I20260925 10:12:44.871237 5339 ts_tablet_manager.cc:1939] T 6aab234abbca44578fd95c6ccbebd7ca P 7ac4363eb0f24649bd2015fa231d68fd: Deleting consensus metadata
I20260925 10:12:44.871814 4789 catalog_manager.cc:4841] TS 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194:35451): tablet 6aab234abbca44578fd95c6ccbebd7ca (table test-table [id=ce103278d22b414eb66c84285853ab1f]) successfully deleted
W20260925 10:12:44.895135 5011 tablet.cc:2369] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:45.005158 5321 raft_consensus.cc:482] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:45.005300 5321 raft_consensus.cc:504] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.005743 5321 leader_election.cc:288] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717), af03440c98e04169a5b14200161663df (127.3.152.195:37707)
I20260925 10:12:45.009917 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.006032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:49957 (local address 127.3.152.193:35717)
0925 10:12:45.006112 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.006114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.006128 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:12:45.006228 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.006229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.006241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.006297 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:45.006533 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.007287 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.008899 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.009455 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.009525 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.009534 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.009809 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.009848 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:12:45.009917 5313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.005989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:48135 (local address 127.3.152.195:37707)
0925 10:12:45.006032 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.006035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.006044 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:45.006166 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.006167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.006195 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.006288 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:45.006533 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.007287 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.008899 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.009448 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.009525 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.009534 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.009809 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.009848 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:45.009958 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.005923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37707 (local address 127.3.152.194:48135)
0925 10:12:45.005990 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:45.006002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:45.006045 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:45.006328 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:45.006329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:45.006331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:45.006438 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.006441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.007394 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.007395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.008797 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.008800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.009596 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.009598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.009726 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.009729 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:45.009787 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 10:12:45.009837 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":59}
I20260925 10:12:45.009966 5344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.005928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35717 (local address 127.3.152.194:49957)
0925 10:12:45.006074 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:45.006085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:45.006149 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:45.006328 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:45.006328 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:45.006331 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:45.006451 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.006452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.007394 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.007395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.008811 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.008813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.009597 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.009598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.009713 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.009718 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:45.009787 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:12:45.009836 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":211,"thread_start_us":65,"threads_started":1}
I20260925 10:12:45.010243 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bcb781c544e8bbcd40b9e6f67cb06" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df" is_pre_election: true
I20260925 10:12:45.010273 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bcb781c544e8bbcd40b9e6f67cb06" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439a87277f14db4a0d5286b2af05a86" is_pre_election: true
I20260925 10:12:45.010370 5245 raft_consensus.cc:2454] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 0.
I20260925 10:12:45.010399 4963 raft_consensus.cc:2454] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 0.
I20260925 10:12:45.010584 5037 leader_election.cc:302] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac4363eb0f24649bd2015fa231d68fd, af03440c98e04169a5b14200161663df; no voters:
I20260925 10:12:45.010746 5321 raft_consensus.cc:2781] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:45.010789 5321 raft_consensus.cc:482] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:45.010810 5321 raft_consensus.cc:3037] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.011478 5321 raft_consensus.cc:504] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.011646 5321 leader_election.cc:288] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 election: Requested vote from peers 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717), af03440c98e04169a5b14200161663df (127.3.152.195:37707)
I20260925 10:12:45.011850 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bcb781c544e8bbcd40b9e6f67cb06" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df"
I20260925 10:12:45.011873 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b28bcb781c544e8bbcd40b9e6f67cb06" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439a87277f14db4a0d5286b2af05a86"
I20260925 10:12:45.011936 4963 raft_consensus.cc:3037] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.011938 5245 raft_consensus.cc:3037] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.012574 5245 raft_consensus.cc:2454] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 1.
I20260925 10:12:45.012782 5037 leader_election.cc:302] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac4363eb0f24649bd2015fa231d68fd, af03440c98e04169a5b14200161663df; no voters:
I20260925 10:12:45.012804 4963 raft_consensus.cc:2454] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 1.
I20260925 10:12:45.012938 5321 raft_consensus.cc:2781] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:45.013118 5321 raft_consensus.cc:686] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 LEADER]: Becoming Leader. State: Replica: 7ac4363eb0f24649bd2015fa231d68fd, State: Running, Role: LEADER
I20260925 10:12:45.013218 5321 consensus_queue.cc:237] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
W20260925 10:12:45.013669 5292 tablet.cc:2369] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:45.013933 4804 catalog_manager.cc:5495] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194). New cstate: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } health_report { overall_health: UNKNOWN } } }
W20260925 10:12:45.070770 5151 tablet.cc:2369] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:12:45 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:45.076256 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.068754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36269 (local address 127.0.0.1:43784)
0925 10:12:45.069008 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:45.069020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:45.069128 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:45.069293 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:45.069293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:45.069712 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:45.069766 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.069769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.070443 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.070444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.071170 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.071173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.071538 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.071539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.071598 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.072189 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:45.072194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:45.073643 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:45.075145 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:45.075152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:45.075154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:45.075392 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:45.075611 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:45.075612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:45.075612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:45.075667 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:45.075901 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:45.075907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:45.075949 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:12:45.076055 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:12:45.076147 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":130,"threads_started":1}
I20260925 10:12:45.076268 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.068855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.3.152.254:36269)
0925 10:12:45.068970 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.068972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.069105 (+ 133us) server_negotiation.cc:407] Connection header received
0925 10:12:45.069192 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.069193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.069204 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.069273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:45.069867 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.070360 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.071272 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.071454 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.071540 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.073729 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:45.073730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:45.073730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:45.073737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:45.075076 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.075478 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.075479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.075479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.075542 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.075736 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.075737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.075737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.075846 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:45.075899 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.076086 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.076196 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":76}
I20260925 10:12:45.078140 4804 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:43784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:12:45.078351 4804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:45.079830 4943 tablet_service.cc:1467] Processing CreateTablet for tablet a104017aafc04791a067869add6df968 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdb39c3152724097adb9ba96f6add2b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:45.079835 5084 tablet_service.cc:1467] Processing CreateTablet for tablet a104017aafc04791a067869add6df968 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdb39c3152724097adb9ba96f6add2b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:45.079905 5225 tablet_service.cc:1467] Processing CreateTablet for tablet a104017aafc04791a067869add6df968 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bdb39c3152724097adb9ba96f6add2b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:45.080044 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a104017aafc04791a067869add6df968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:45.080045 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a104017aafc04791a067869add6df968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:45.080044 5225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a104017aafc04791a067869add6df968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:45.081455 5319 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: Bootstrap starting.
I20260925 10:12:45.081682 5315 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: Bootstrap starting.
I20260925 10:12:45.081839 5319 tablet_bootstrap.cc:654] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:45.082124 5315 tablet_bootstrap.cc:654] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:45.082413 5319 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: No bootstrap required, opened a new log
I20260925 10:12:45.082455 5319 ts_tablet_manager.cc:1397] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:45.082461 5316 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: Bootstrap starting.
I20260925 10:12:45.082569 5319 raft_consensus.cc:348] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.082653 5319 raft_consensus.cc:374] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:45.082679 5319 raft_consensus.cc:729] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439a87277f14db4a0d5286b2af05a86, State: Initialized, Role: FOLLOWER
I20260925 10:12:45.082728 5319 consensus_queue.cc:260] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.082844 5316 tablet_bootstrap.cc:654] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:45.082857 5319 ts_tablet_manager.cc:1428] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:45.082917 5323 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.083009 5323 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.083384 5315 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: No bootstrap required, opened a new log
I20260925 10:12:45.083431 5315 ts_tablet_manager.cc:1397] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:12:45.083465 5316 tablet_bootstrap.cc:492] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: No bootstrap required, opened a new log
I20260925 10:12:45.083506 5316 ts_tablet_manager.cc:1397] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:45.083567 5315 raft_consensus.cc:348] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.083616 5315 raft_consensus.cc:374] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:45.083628 5315 raft_consensus.cc:729] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af03440c98e04169a5b14200161663df, State: Initialized, Role: FOLLOWER
I20260925 10:12:45.083662 5315 consensus_queue.cc:260] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.083647 5316 raft_consensus.cc:348] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.083731 5316 raft_consensus.cc:374] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:45.083745 5315 ts_tablet_manager.cc:1428] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:45.083755 5316 raft_consensus.cc:729] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac4363eb0f24649bd2015fa231d68fd, State: Initialized, Role: FOLLOWER
I20260925 10:12:45.083801 5316 consensus_queue.cc:260] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.083889 5316 ts_tablet_manager.cc:1428] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:45.083840 5320 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.083940 5320 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.083931 5321 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.084055 5321 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.180303 5321 raft_consensus.cc:482] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:45.180413 5321 raft_consensus.cc:504] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.180619 5321 leader_election.cc:288] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717), af03440c98e04169a5b14200161663df (127.3.152.195:37707)
I20260925 10:12:45.180958 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a104017aafc04791a067869add6df968" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439a87277f14db4a0d5286b2af05a86" is_pre_election: true
I20260925 10:12:45.180958 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a104017aafc04791a067869add6df968" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df" is_pre_election: true
I20260925 10:12:45.181077 5245 raft_consensus.cc:2454] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 0.
I20260925 10:12:45.181102 4963 raft_consensus.cc:2454] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 0.
I20260925 10:12:45.181264 5037 leader_election.cc:302] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac4363eb0f24649bd2015fa231d68fd, af03440c98e04169a5b14200161663df; no voters:
I20260925 10:12:45.181385 5321 raft_consensus.cc:2781] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:45.181427 5321 raft_consensus.cc:482] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:45.181447 5321 raft_consensus.cc:3037] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.182127 5321 raft_consensus.cc:504] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.182277 5321 leader_election.cc:288] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 election: Requested vote from peers 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717), af03440c98e04169a5b14200161663df (127.3.152.195:37707)
I20260925 10:12:45.182438 4963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a104017aafc04791a067869add6df968" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4439a87277f14db4a0d5286b2af05a86"
I20260925 10:12:45.182461 5245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a104017aafc04791a067869add6df968" candidate_uuid: "7ac4363eb0f24649bd2015fa231d68fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af03440c98e04169a5b14200161663df"
I20260925 10:12:45.182526 5245 raft_consensus.cc:3037] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.182525 4963 raft_consensus.cc:3037] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:45.183293 5245 raft_consensus.cc:2454] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 1.
I20260925 10:12:45.183293 4963 raft_consensus.cc:2454] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac4363eb0f24649bd2015fa231d68fd in term 1.
I20260925 10:12:45.183471 5037 leader_election.cc:302] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac4363eb0f24649bd2015fa231d68fd, af03440c98e04169a5b14200161663df; no voters:
I20260925 10:12:45.183615 5321 raft_consensus.cc:2781] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:45.183683 5321 raft_consensus.cc:686] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 LEADER]: Becoming Leader. State: Replica: 7ac4363eb0f24649bd2015fa231d68fd, State: Running, Role: LEADER
I20260925 10:12:45.183722 5321 consensus_queue.cc:237] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } }
I20260925 10:12:45.184051 5345 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: TxnStatusTablet state changed. Reason: New leader 7ac4363eb0f24649bd2015fa231d68fd. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.184130 5345 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:45.184314 5357 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:12:45.184444 5357 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:12:45.184408 4805 catalog_manager.cc:5495] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194). New cstate: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:45.203109 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.200043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35451 (local address 127.0.0.1:51226)
0925 10:12:45.200170 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:45.200181 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:45.200240 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:45.200415 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:45.200415 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:45.200416 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:45.200464 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.200466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.201369 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.201370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.202018 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.202021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.202483 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.202484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.202532 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.202548 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:45.202938 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:45.202951 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:12:45.203003 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:12:45.203045 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:12:45.203109 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.200097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.3.152.194:35451)
0925 10:12:45.200156 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.200158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.200216 (+ 58us) server_negotiation.cc:407] Connection header received
0925 10:12:45.200307 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.200309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.200330 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.200413 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:45.200505 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.201275 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.202102 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.202405 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.202474 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.202704 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:45.202846 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:45.202924 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.203009 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.203028 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:12:45.204413 5245 raft_consensus.cc:1261] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Refusing update from remote peer 7ac4363eb0f24649bd2015fa231d68fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:45.204430 4963 raft_consensus.cc:1261] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Refusing update from remote peer 7ac4363eb0f24649bd2015fa231d68fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:45.204702 5345 consensus_queue.cc:1035] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:45.204833 5321 consensus_queue.cc:1035] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:45.205446 5323 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: TxnStatusTablet state changed. Reason: New leader 7ac4363eb0f24649bd2015fa231d68fd. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.205443 5320 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: TxnStatusTablet state changed. Reason: New leader 7ac4363eb0f24649bd2015fa231d68fd. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.205539 5323 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.205556 5320 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.206482 5345 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.206576 5345 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:45.206489 5321 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.206611 5321 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 7ac4363eb0f24649bd2015fa231d68fd: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:45.206640 5330 mvcc.cc:205] Tried to move back new op lower bound from 7333196452674740224 to 7333196452592943104. Current Snapshot: MvccSnapshot[applied={T|T < 7333196452674740224}]
I20260925 10:12:45.206715 5329 mvcc.cc:205] Tried to move back new op lower bound from 7333196452674740224 to 7333196452592943104. Current Snapshot: MvccSnapshot[applied={T|T < 7333196452674740224}]
I20260925 10:12:45.206797 5320 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.206864 5320 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P af03440c98e04169a5b14200161663df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.207516 5323 tablet_replica.cc:439] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ac4363eb0f24649bd2015fa231d68fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 } } peers { permanent_uuid: "7ac4363eb0f24649bd2015fa231d68fd" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35451 } } peers { permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 } } }
I20260925 10:12:45.207599 5323 tablet_replica.cc:442] T a104017aafc04791a067869add6df968 P 4439a87277f14db4a0d5286b2af05a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:45.210927 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.208467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.3.152.194:35451)
0925 10:12:45.208501 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.208503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.208532 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:12:45.208610 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.208611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.208622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.208670 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:45.208791 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.209274 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.210340 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.210519 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.210582 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.210740 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:45.210768 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:45.210807 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.210861 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.210876 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:12:45.217653 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.213797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:50279 (local address 127.3.152.194:35451)
0925 10:12:45.214905 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.214907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.214915 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:12:45.214931 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.214932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.214940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.215013 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:45.215167 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.215612 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.216945 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.217124 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.217200 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.217400 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:45.217431 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:45.217489 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.217548 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.217570 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089}
I20260925 10:12:45.217759 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.213636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35451 (local address 127.3.152.194:50279)
0925 10:12:45.213802 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:45.213815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:45.213868 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:45.215030 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:45.215032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:45.215034 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:45.215093 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.215094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.215712 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.215713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.216840 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:45.216843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.217198 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.217199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:45.217284 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.217309 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:45.217490 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:45.217492 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:45.217545 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:12:45.217592 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260925 10:12:45.220170 5245 raft_consensus.cc:1261] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Refusing update from remote peer 7ac4363eb0f24649bd2015fa231d68fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:45.220340 4963 raft_consensus.cc:1261] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Refusing update from remote peer 7ac4363eb0f24649bd2015fa231d68fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:45.220405 5321 consensus_queue.cc:1035] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "af03440c98e04169a5b14200161663df" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:45.220538 5362 consensus_queue.cc:1035] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4439a87277f14db4a0d5286b2af05a86" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:45.233062 4805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:45.233173 4805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:45.233928 4805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a92ceeea9394c97ae97f925ff2b3011: Sending DeleteTablet for 3 replicas of tablet b28bcb781c544e8bbcd40b9e6f67cb06
I20260925 10:12:45.234212 4943 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:38950
I20260925 10:12:45.234328 5225 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40216
I20260925 10:12:45.234354 5340 tablet_replica.cc:330] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: stopping tablet replica
I20260925 10:12:45.234408 5083 tablet_service.cc:1514] Processing DeleteTablet for tablet b28bcb781c544e8bbcd40b9e6f67cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:51224
I20260925 10:12:45.234438 5340 raft_consensus.cc:2229] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:45.234483 5340 raft_consensus.cc:2258] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:45.234513 5338 tablet_replica.cc:330] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: stopping tablet replica
I20260925 10:12:45.234516 5339 tablet_replica.cc:330] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: stopping tablet replica
I20260925 10:12:45.234568 5338 raft_consensus.cc:2229] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:45.234613 5338 raft_consensus.cc:2258] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:45.234575 5339 raft_consensus.cc:2229] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:45.234745 5339 raft_consensus.cc:2258] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:45.234961 5338 ts_tablet_manager.cc:1905] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:45.235528 5339 ts_tablet_manager.cc:1905] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:45.237143 5338 ts_tablet_manager.cc:1918] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:45.237214 5338 log.cc:1196] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/b28bcb781c544e8bbcd40b9e6f67cb06
I20260925 10:12:45.237371 5340 ts_tablet_manager.cc:1905] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:45.237418 5338 ts_tablet_manager.cc:1939] T b28bcb781c544e8bbcd40b9e6f67cb06 P af03440c98e04169a5b14200161663df: Deleting consensus metadata
I20260925 10:12:45.237799 5339 ts_tablet_manager.cc:1918] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:45.237851 5339 log.cc:1196] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/b28bcb781c544e8bbcd40b9e6f67cb06
I20260925 10:12:45.237825 4791 catalog_manager.cc:4841] TS af03440c98e04169a5b14200161663df (127.3.152.195:37707): tablet b28bcb781c544e8bbcd40b9e6f67cb06 (table test-table [id=3d5a8676c9d0488eb642b236475a5b8a]) successfully deleted
I20260925 10:12:45.238107 5339 ts_tablet_manager.cc:1939] T b28bcb781c544e8bbcd40b9e6f67cb06 P 7ac4363eb0f24649bd2015fa231d68fd: Deleting consensus metadata
I20260925 10:12:45.238415 4789 catalog_manager.cc:4841] TS 7ac4363eb0f24649bd2015fa231d68fd (127.3.152.194:35451): tablet b28bcb781c544e8bbcd40b9e6f67cb06 (table test-table [id=3d5a8676c9d0488eb642b236475a5b8a]) successfully deleted
I20260925 10:12:45.239046 5340 ts_tablet_manager.cc:1918] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:45.239112 5340 log.cc:1196] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/b28bcb781c544e8bbcd40b9e6f67cb06
I20260925 10:12:45.239358 5340 ts_tablet_manager.cc:1939] T b28bcb781c544e8bbcd40b9e6f67cb06 P 4439a87277f14db4a0d5286b2af05a86: Deleting consensus metadata
I20260925 10:12:45.239713 4792 catalog_manager.cc:4841] TS 4439a87277f14db4a0d5286b2af05a86 (127.3.152.193:35717): tablet b28bcb781c544e8bbcd40b9e6f67cb06 (table test-table [id=3d5a8676c9d0488eb642b236475a5b8a]) successfully deleted
Sep 25 10:12:45 dist-test-slave-dzwq krb5kdc[4760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:45.252768 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.235550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.3.152.193:35717)
0925 10:12:45.235612 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.235614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.240209 (+ 4595us) server_negotiation.cc:407] Connection header received
0925 10:12:45.240293 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.240294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.240305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.240357 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:45.241192 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.242040 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.243296 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.243647 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.243697 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.248299 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:45.248312 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:45.248313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:45.248323 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:45.251210 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.251595 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.251600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.251600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.251674 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.251883 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.251884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.251884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.252099 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:45.252255 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.252486 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.252598 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:12:45.253502 5004 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38972
I20260925 10:12:45.263639 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:45.254699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.3.152.254:36269)
0925 10:12:45.254748 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:45.254750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:45.255007 (+ 257us) server_negotiation.cc:407] Connection header received
0925 10:12:45.255125 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:45.255127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:45.255141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:45.255208 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:45.255931 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.256878 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.258048 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:45.258436 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:45.258508 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:45.260288 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:45.260290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:45.260290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:45.260300 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:45.262237 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.262698 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.262699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.262699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.262779 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:45.263017 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:45.263018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:45.263019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:45.263186 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:45.263295 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:45.263445 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:12:45.263536 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 10:12:45.263867 4805 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43788
I20260925 10:12:45.264742 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4878
I20260925 10:12:45.274796 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5019
I20260925 10:12:45.279796 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5159
I20260925 10:12:45.283980 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 4773
2026-09-25T10:12:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2366 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:45 dist-test-slave-dzwq krb5kdc[5376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:45 dist-test-slave-dzwq krb5kdc[5376](info): set up 2 sockets
Sep 25 10:12:45 dist-test-slave-dzwq krb5kdc[5376](info): commencing operation
krb5kdc: starting...
W20260925 10:12:46.376456 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:46 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:46.507748 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41259
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:46.593923 5389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:46.594177 5389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:46.594202 5389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:46.595467 5389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:46.595520 5389 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:12:46.595534 5389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:46.595547 5389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:46.595561 5389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:46.595573 5389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:46.597245 5389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41259
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:46.597568 5389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:46.597848 5389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:46.603068 5397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:46.603082 5394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:46.603243 5395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:46.603433 5389 server_base.cc:1034] running on GCE node
I20260925 10:12:46.603699 5389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:46.604020 5389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:46.605206 5389 hybrid_clock.cc:648] HybridClock initialized: now 1790331166605187 us; error 31 us; skew 500 ppm
Sep 25 10:12:46 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:46.608237 5389 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:46.608783 5389 webserver.cc:460] Webserver started at http://127.3.152.254:44153/ using document root <none> and password file <none>
I20260925 10:12:46.609025 5389 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:46.609097 5389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:46.609244 5389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:46.610385 5389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "0ffbd894eda60b9dc7312893e54c25d1"
server_key_iv: "46f9801bd1a8bb9922aca20be89d3da1"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.610787 5389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "0ffbd894eda60b9dc7312893e54c25d1"
server_key_iv: "46f9801bd1a8bb9922aca20be89d3da1"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.612416 5389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:46.613346 5404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:46.613559 5389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:46.613705 5389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "0ffbd894eda60b9dc7312893e54c25d1"
server_key_iv: "46f9801bd1a8bb9922aca20be89d3da1"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.613799 5389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:46.657552 5389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:46.660099 5389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:46.660274 5389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:46.665119 5389 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41259
I20260925 10:12:46.665258 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41259 every 8 connection(s)
I20260925 10:12:46.665562 5389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:46.666280 5468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:46.668718 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Bootstrap starting.
I20260925 10:12:46.669344 5468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:46.669662 5468 log.cc:824] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:46.670406 5468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: No bootstrap required, opened a new log
I20260925 10:12:46.672785 5468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } }
I20260925 10:12:46.672948 5468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:46.672971 5468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717b7fb79dbe4fc1b1d76bfcab55c4d3, State: Initialized, Role: FOLLOWER
I20260925 10:12:46.673161 5468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } }
I20260925 10:12:46.673241 5468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:46.673264 5468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:46.673319 5468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:46.673861 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5389
I20260925 10:12:46.673934 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:46.673995 5468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } }
I20260925 10:12:46.674139 3683 external_mini_cluster.cc:1428] Setting key 25d1f2bec78c21b7ed1b02b9cf660ffb
I20260925 10:12:46.674141 5468 leader_election.cc:302] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717b7fb79dbe4fc1b1d76bfcab55c4d3; no voters:
I20260925 10:12:46.674353 5468 leader_election.cc:288] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:46.674387 5471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:46.674500 5471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [term 1 LEADER]: Becoming Leader. State: Replica: 717b7fb79dbe4fc1b1d76bfcab55c4d3, State: Running, Role: LEADER
I20260925 10:12:46.674597 5471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } }
I20260925 10:12:46.674682 5468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:46.674917 5471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717b7fb79dbe4fc1b1d76bfcab55c4d3. Latest consensus state: current_term: 1 leader_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } } }
I20260925 10:12:46.675102 5471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:46.674932 5472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717b7fb79dbe4fc1b1d76bfcab55c4d3" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41259 } } }
I20260925 10:12:46.675627 5472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:46.675783 5480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:46.676365 5480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:46.678028 5480 catalog_manager.cc:1347] Generated new cluster ID: 68f7dbccc69d4dfd8840e3967476d8ba
I20260925 10:12:46.678085 5480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:46 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:46.685947 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:46.674942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.3.152.254:41259)
0925 10:12:46.675349 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:46.675360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:46.675397 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:12:46.675528 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:46.675537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:46.675600 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:46.675783 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:46.676877 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.678208 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.679009 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.679477 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.679601 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:46.682037 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:46.682041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:46.682041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:46.682055 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:46.684298 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.684688 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.684692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.684692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.684811 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.685029 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.685030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.685030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.685293 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:46.685467 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:46.685566 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:12:46.685720 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":74,"threads_started":1}
I20260925 10:12:46.692930 5480 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:46.693550 5480 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:46.699505 5480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Generated new TSK 0
I20260925 10:12:46.699724 5480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:46.731918 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41259
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:46.835316 5494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:46.835542 5494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:46.835572 5494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:46.836871 5494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:46.836923 5494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:46.836946 5494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:12:46.836958 5494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:46.836977 5494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:46.838537 5494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:46.838809 5494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:46.839087 5494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:46.845466 5502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:46.845458 5499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:46.845577 5494 server_base.cc:1034] running on GCE node
W20260925 10:12:46.845458 5500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:46.845953 5494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:46.846258 5494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:46.847434 5494 hybrid_clock.cc:648] HybridClock initialized: now 1790331166847421 us; error 33 us; skew 500 ppm
Sep 25 10:12:46 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:46.850241 5494 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:46.850852 5494 webserver.cc:460] Webserver started at http://127.3.152.193:37101/ using document root <none> and password file <none>
I20260925 10:12:46.851096 5494 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:46.851199 5494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:46.851356 5494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:46.852566 5494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "a218faf652494a378fd285cc2b8b5c9c"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "2c2d74491515e776f99262c61ed4cfec"
server_key_iv: "ea564addc3289daaf7a42a04681dff9e"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.852945 5494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "a218faf652494a378fd285cc2b8b5c9c"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "2c2d74491515e776f99262c61ed4cfec"
server_key_iv: "ea564addc3289daaf7a42a04681dff9e"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.854449 5494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:46.855453 5509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:46.855792 5494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:46.855923 5494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "a218faf652494a378fd285cc2b8b5c9c"
format_stamp: "Formatted at 2026-09-25 10:12:46 on dist-test-slave-dzwq"
server_key: "2c2d74491515e776f99262c61ed4cfec"
server_key_iv: "ea564addc3289daaf7a42a04681dff9e"
server_key_version: "encryptionkey@0"
I20260925 10:12:46.856068 5494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:46.873075 5494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:46.876389 5494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:46.876649 5494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:46.877249 5494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:46.877305 5494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:46.877329 5494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:46.877363 5494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:12:46 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:46.885480 5494 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35053
I20260925 10:12:46.885571 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35053 every 8 connection(s)
I20260925 10:12:46.885967 5494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:46.888325 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5494
I20260925 10:12:46.888460 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:46.888626 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:46.877755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:43053 (local address 127.3.152.254:41259)
0925 10:12:46.877798 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:46.877801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:46.878031 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:12:46.878601 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:46.878603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:46.878619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:46.878683 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:46.879455 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.880215 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.881320 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.881538 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.881626 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:46.885309 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:46.885311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:46.885311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:46.885321 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:46.887073 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.887521 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.887523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.887523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.887589 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.887868 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.887868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.887869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.888063 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:46.888145 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:46.888439 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 10:12:46.888531 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:12:46.888685 3683 external_mini_cluster.cc:1428] Setting key 06075e633f3fcd5cd3b848ec34fee5c6
I20260925 10:12:46.888849 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:46.877743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.193:43053)
0925 10:12:46.877904 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:46.877941 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:46.878476 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:46.878769 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:46.878773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:46.879144 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:46.879349 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:46.879354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.880323 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.880324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:46.881195 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:46.881198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.881639 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.881640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:46.881821 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:46.882472 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:46.882487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:46.885186 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:46.887187 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:46.887194 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:46.887196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:46.887436 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:46.887663 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:46.887663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:46.887664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:46.887788 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:46.888136 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:46.888143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:46.888298 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:12:46.888414 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 10:12:46.888601 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260925 10:12:46.891130 5626 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41259
I20260925 10:12:46.891261 5626 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:46.891467 5626 heartbeater.cc:511] Master 127.3.152.254:41259 requested a full tablet report, sending...
I20260925 10:12:46.892223 5421 ts_manager.cc:194] Registered new tserver with Master: a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053)
I20260925 10:12:46.892866 5421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:43053
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
I20260925 10:12:46.904383 5633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:46.890903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.193:33571)
0925 10:12:46.897437 (+ 6534us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:46.897449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:46.897567 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:46.897738 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:46.897739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:46.897945 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:46.898002 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:46.898003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.898856 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.898858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:46.899430 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:46.899432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.899819 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.899820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:46.899902 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:46.900524 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:46.900529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:46.900921 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:46.903090 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:46.903091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:46.903092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:46.903315 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:46.903585 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:46.903586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:46.903587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:46.903660 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:46.903943 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:46.903944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:46.904034 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 10:12:46.904144 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:12:46.904269 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6497,"thread_start_us":83,"threads_started":1}
I20260925 10:12:46.904378 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:46.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:33571 (local address 127.3.152.254:41259)
0925 10:12:46.890996 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:46.890998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:46.897549 (+ 6551us) server_negotiation.cc:407] Connection header received
0925 10:12:46.897652 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:46.897653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:46.897663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:46.897731 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:46.898092 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.898744 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.899527 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:46.899728 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:46.899818 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:46.901047 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:46.901049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:46.901049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:46.901058 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:46.902983 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.903415 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.903417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.903417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.903490 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:46.903731 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:46.903732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:46.903732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:46.903869 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:46.903981 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:46.904172 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:12:46.904269 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:46.919759 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41259
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:47.005868 5635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:47.006098 5635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:47.006125 5635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:47.007288 5635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:47.007329 5635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:47.007350 5635 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:12:47.007364 5635 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:47.007380 5635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:47.008874 5635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:47.009089 5635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:47.009326 5635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:47.014807 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:47.014807 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:47.014808 5643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:47.015089 5635 server_base.cc:1034] running on GCE node
I20260925 10:12:47.015318 5635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:47.015592 5635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:47.016779 5635 hybrid_clock.cc:648] HybridClock initialized: now 1790331167016768 us; error 30 us; skew 500 ppm
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:47.019696 5635 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:47.020241 5635 webserver.cc:460] Webserver started at http://127.3.152.194:45195/ using document root <none> and password file <none>
I20260925 10:12:47.020531 5635 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:47.020609 5635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:47.020766 5635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:47.021871 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "500e298a1c114938b317de53f4fb675d"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "fd5d5a6a5a52ee80a6276f9ed64cbea0"
server_key_iv: "57370f887f918162f438e8c4063f5858"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.022245 5635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "500e298a1c114938b317de53f4fb675d"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "fd5d5a6a5a52ee80a6276f9ed64cbea0"
server_key_iv: "57370f887f918162f438e8c4063f5858"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.023747 5635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:47.024776 5650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.025003 5635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:47.025122 5635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "500e298a1c114938b317de53f4fb675d"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "fd5d5a6a5a52ee80a6276f9ed64cbea0"
server_key_iv: "57370f887f918162f438e8c4063f5858"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.025216 5635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:47.036016 5635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:47.038583 5635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:47.038763 5635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:47.039326 5635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:47.039392 5635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.039433 5635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:47.039481 5635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.046792 5635 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:38669
I20260925 10:12:47.046883 5765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:38669 every 8 connection(s)
I20260925 10:12:47.047230 5635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:47.051079 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.040005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53527 (local address 127.3.152.254:41259)
0925 10:12:47.040085 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.040088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.040236 (+ 148us) server_negotiation.cc:407] Connection header received
0925 10:12:47.040870 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.040872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.040887 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.040973 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.041880 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.042736 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.044444 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.044765 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.044870 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.048021 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.048022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.048023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.048032 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.049285 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.049735 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.049737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.049738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.049808 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.050080 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.050081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.050082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.050259 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.050329 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.050861 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.050982 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:12:47.051589 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.039889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.194:53527)
0925 10:12:47.040103 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.040144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.040760 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.041049 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.041053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.041435 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:47.041748 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.041754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.042867 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.042869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.044342 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.044345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.044877 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.044878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.044986 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.045821 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:47.045833 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:47.047862 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:47.049362 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.049368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.049370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.049609 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.049880 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.049881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.049882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.050005 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.050383 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:47.050390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:47.050603 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.050793 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.050987 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260925 10:12:47.052196 5766 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41259
I20260925 10:12:47.052345 5766 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:47.052546 5766 heartbeater.cc:511] Master 127.3.152.254:41259 requested a full tablet report, sending...
I20260925 10:12:47.052986 5421 ts_manager.cc:194] Registered new tserver with Master: 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669)
I20260925 10:12:47.053464 5421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:53527
I20260925 10:12:47.055514 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5635
I20260925 10:12:47.055627 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:47.055825 3683 external_mini_cluster.cc:1428] Setting key d77770407078c4aa8c0d45b4fc66948a
I20260925 10:12:47.059468 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.053362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.194:46365)
0925 10:12:47.053532 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.053544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.053605 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.053737 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.053738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.053940 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:47.053994 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.053996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.054683 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.054684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.055198 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.055199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.055592 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.055593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.055670 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.056250 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:47.056255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:47.056636 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:47.058198 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.058199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.058199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.058376 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.058582 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.058583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.058583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.058641 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.059048 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:47.059049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:47.059179 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.059273 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.059378 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260925 10:12:47.059547 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:46365 (local address 127.3.152.254:41259)
0925 10:12:47.053476 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.053478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.053590 (+ 112us) server_negotiation.cc:407] Connection header received
0925 10:12:47.053664 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.053666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.053677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.053732 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.054074 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.054574 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.055297 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.055503 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.055590 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.056730 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.056731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.056732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.056739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.058090 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.058450 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.058452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.058453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.058510 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.058718 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.058719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.058719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.058920 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.058991 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.059315 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.059407 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:47.084260 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41259
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:47.171970 5775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:47.172264 5775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:47.172298 5775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:47.173638 5775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:47.173694 5775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:47.173722 5775 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 10:12:47.173736 5775 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:12:47.173758 5775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:47.175585 5775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:47.175935 5775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:47.176326 5775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:47.182410 5783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:47.182423 5780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:47.182509 5775 server_base.cc:1034] running on GCE node
W20260925 10:12:47.182438 5781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:47.182964 5775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:47.183239 5775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:47.184429 5775 hybrid_clock.cc:648] HybridClock initialized: now 1790331167184404 us; error 37 us; skew 500 ppm
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:47.187325 5775 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:47.187852 5775 webserver.cc:460] Webserver started at http://127.3.152.195:35373/ using document root <none> and password file <none>
I20260925 10:12:47.188169 5775 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:47.188242 5775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:47.188406 5775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:47.189687 5775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "3b216211c5764fec93f94282d6d5169f"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "0a4df15f0cef3985f6a5a58c549871ef"
server_key_iv: "22b6a60dc9ce66ef767ef89961314c6e"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.190063 5775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "3b216211c5764fec93f94282d6d5169f"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "0a4df15f0cef3985f6a5a58c549871ef"
server_key_iv: "22b6a60dc9ce66ef767ef89961314c6e"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.191568 5775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:47.192497 5790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.192713 5775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:47.192844 5775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "3b216211c5764fec93f94282d6d5169f"
format_stamp: "Formatted at 2026-09-25 10:12:47 on dist-test-slave-dzwq"
server_key: "0a4df15f0cef3985f6a5a58c549871ef"
server_key_iv: "22b6a60dc9ce66ef767ef89961314c6e"
server_key_version: "encryptionkey@0"
I20260925 10:12:47.192924 5775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:47.201661 5775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:47.203640 5775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:47.203819 5775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:47.204432 5775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:47.204492 5775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.204535 5775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:47.204561 5775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:47.211675 5775 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:44249
I20260925 10:12:47.211742 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:44249 every 8 connection(s)
I20260925 10:12:47.212143 5775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:47.214433 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.204834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53467 (local address 127.3.152.254:41259)
0925 10:12:47.204913 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.204916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.205114 (+ 198us) server_negotiation.cc:407] Connection header received
0925 10:12:47.205687 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.205688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.205703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.205754 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.206564 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.207396 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.208783 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.209000 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.209092 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.211524 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.211526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.211526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.211536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.213211 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.213614 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.213614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.213615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.213669 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.213867 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.213867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.213868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.213976 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.214027 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.214300 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.214367 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:47.214592 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.204840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.195:53467)
0925 10:12:47.204993 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.205029 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.205565 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.205859 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.205863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.206225 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:47.206450 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.206455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.207509 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.207510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.208664 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.208667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.209120 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.209121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.209258 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.209847 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:47.209855 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:47.211398 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:47.213306 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.213311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.213313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.213540 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.213714 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.213715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.213715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.213813 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.214033 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:47.214038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:47.214169 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.214256 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.214386 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260925 10:12:47.215132 5907 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41259
I20260925 10:12:47.215224 5907 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:47.215425 5907 heartbeater.cc:511] Master 127.3.152.254:41259 requested a full tablet report, sending...
I20260925 10:12:47.215845 5420 ts_manager.cc:194] Registered new tserver with Master: 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:47.216378 5420 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:53467
I20260925 10:12:47.219758 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 5775
I20260925 10:12:47.219863 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:47.220145 3683 external_mini_cluster.cc:1428] Setting key 2067db7526c513afdc8f8fa67eb25bc5
I20260925 10:12:47.221101 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:12:47.222496 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.216263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:46791 (local address 127.3.152.254:41259)
0925 10:12:47.216307 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.216309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.216384 (+ 75us) server_negotiation.cc:407] Connection header received
0925 10:12:47.216451 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.216452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.216462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.216512 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.216888 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.217560 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.218330 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.218511 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.218566 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.219593 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.219594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.219595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.219603 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.221380 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.221792 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.221793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.221794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.221848 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.222051 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.222051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.222052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.222155 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.222205 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.222374 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.222434 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:47.222657 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.216198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.3.152.195:46791)
0925 10:12:47.216323 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.216333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.216395 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.216531 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.216532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.216758 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:47.216830 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.216831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.217733 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.217734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.218252 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.218254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.218579 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.218580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.218637 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.219173 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:47.219178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:47.219482 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:47.221453 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.221454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.221454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.221702 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.221914 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.221915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.221916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.221990 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.222219 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:47.222220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:47.222279 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.222359 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.222484 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:47.247310 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.239101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.3.152.254:41259)
0925 10:12:47.239176 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.239178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.239190 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:12:47.239243 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.239244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.239262 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.239329 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.239991 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.240504 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.241336 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.241540 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.241620 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.244242 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.244243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.244244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.244254 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.245880 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.246261 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.246262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.246263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.246323 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.246728 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.246730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.246730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.246860 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.246957 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.247117 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.247202 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:12:47.249468 5420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:47.251283 5420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:47.258445 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38669 (local address 127.0.0.1:39562)
0925 10:12:47.254387 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.254438 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.254514 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.254919 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.254921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.254923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.254990 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.254992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.255999 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.256001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.257276 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.257279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.258080 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.258176 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258193 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.258291 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.258350 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":69,"threads_started":1}
I20260925 10:12:47.258451 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.3.152.194:38669)
0925 10:12:47.254658 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.254661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.254670 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:12:47.254705 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.254706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.254718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.254833 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.255068 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.255865 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.257386 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.257970 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258076 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258145 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.258310 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.258350 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":87,"threads_started":1}
I20260925 10:12:47.258662 5924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:44249 (local address 127.0.0.1:46528)
0925 10:12:47.254463 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.254473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.254524 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.254882 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.254886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.254889 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.254973 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.254975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.256381 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.256382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.257445 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.257447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.258374 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.258451 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258476 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.258510 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.258555 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260925 10:12:47.259099 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.3.152.195:44249)
0925 10:12:47.254696 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.254702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.254712 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:47.254755 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.254758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.254774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.255054 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.255115 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.256281 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.257553 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.258272 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258881 (+ 609us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.258969 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.259008 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260925 10:12:47.259333 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 863ce6fe628c48e68493859cd46f6eed (DEFAULT_TABLE table=test-table [id=d7e112f0008f4771aefddee62fa5008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.259735 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863ce6fe628c48e68493859cd46f6eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.259686 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 863ce6fe628c48e68493859cd46f6eed (DEFAULT_TABLE table=test-table [id=d7e112f0008f4771aefddee62fa5008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.260066 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863ce6fe628c48e68493859cd46f6eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.260174 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.3.152.193:35053)
0925 10:12:47.254569 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.254575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.254839 (+ 264us) server_negotiation.cc:407] Connection header received
0925 10:12:47.254890 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.254892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.254907 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.254981 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.255161 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.256325 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.257507 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.258266 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258343 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258400 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.260004 (+ 1604us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.260054 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":80,"threads_started":1}
I20260925 10:12:47.260376 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.254166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.0.0.1:49182)
0925 10:12:47.254465 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.254683 (+ 218us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.254746 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.255050 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.255051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.255052 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.255084 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.255085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.256415 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.256416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.257402 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.257404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.258428 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.258429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.258480 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.258483 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.258703 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.260311 (+ 1608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":4,"thread_start_us":122,"threads_started":1}
I20260925 10:12:47.263473 5931 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Bootstrap starting.
I20260925 10:12:47.263792 5932 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Bootstrap starting.
I20260925 10:12:47.264477 5932 tablet_bootstrap.cc:654] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.264767 5931 tablet_bootstrap.cc:654] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.264639 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 863ce6fe628c48e68493859cd46f6eed (DEFAULT_TABLE table=test-table [id=d7e112f0008f4771aefddee62fa5008f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.264875 5932 log.cc:824] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:47.265030 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863ce6fe628c48e68493859cd46f6eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.265161 5931 log.cc:824] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:47.265811 5932 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: No bootstrap required, opened a new log
I20260925 10:12:47.265890 5932 ts_tablet_manager.cc:1397] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:47.265954 5931 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: No bootstrap required, opened a new log
I20260925 10:12:47.266023 5931 ts_tablet_manager.cc:1397] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 10:12:47.267401 5931 raft_consensus.cc:348] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.267570 5931 raft_consensus.cc:374] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.267616 5931 raft_consensus.cc:729] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.267782 5931 consensus_queue.cc:260] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.268110 5907 heartbeater.cc:503] Master 127.3.152.254:41259 was elected leader, sending a full tablet report...
I20260925 10:12:47.267942 5932 raft_consensus.cc:348] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.268167 5932 raft_consensus.cc:374] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.268219 5932 raft_consensus.cc:729] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.268335 5932 consensus_queue.cc:260] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.268452 5931 ts_tablet_manager.cc:1428] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:12:47.268607 5932 ts_tablet_manager.cc:1428] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 10:12:47.268653 5766 heartbeater.cc:503] Master 127.3.152.254:41259 was elected leader, sending a full tablet report...
I20260925 10:12:47.268831 5937 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Bootstrap starting.
I20260925 10:12:47.269493 5937 tablet_bootstrap.cc:654] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.269866 5937 log.cc:824] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:47.270586 5937 tablet_bootstrap.cc:492] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: No bootstrap required, opened a new log
I20260925 10:12:47.270643 5937 ts_tablet_manager.cc:1397] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:47.272054 5937 raft_consensus.cc:348] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.272203 5937 raft_consensus.cc:374] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.272226 5937 raft_consensus.cc:729] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.272303 5937 consensus_queue.cc:260] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.272493 5937 ts_tablet_manager.cc:1428] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:12:47.272567 5626 heartbeater.cc:503] Master 127.3.152.254:41259 was elected leader, sending a full tablet report...
I20260925 10:12:47.278050 5935 raft_consensus.cc:482] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:47.278163 5935 raft_consensus.cc:504] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.278577 5935 leader_election.cc:288] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053)
I20260925 10:12:47.281852 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.278783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38669 (local address 127.3.152.195:34437)
0925 10:12:47.278833 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.278846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.278898 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.279028 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.279029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.279030 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.279097 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.279099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.279784 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.279786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.280639 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.280641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.281437 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.281438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.281537 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.281543 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.281580 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.281633 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":25}
I20260925 10:12:47.281831 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.278909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:52981 (local address 127.3.152.193:35053)
0925 10:12:47.278965 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.278967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.279098 (+ 131us) server_negotiation.cc:407] Connection header received
0925 10:12:47.279186 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.279187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.279195 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.279248 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.279406 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.279880 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.280814 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.281409 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.281489 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.281498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.281670 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.281709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:47.282163 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.278790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:34437 (local address 127.3.152.194:38669)
0925 10:12:47.278837 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.278840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.278880 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:12:47.278962 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.278963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.278971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.279024 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.279183 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.279681 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.280732 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.281321 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.282003 (+ 682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.282015 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.282037 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.282084 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:12:47.282486 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863ce6fe628c48e68493859cd46f6eed" candidate_uuid: "3b216211c5764fec93f94282d6d5169f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d" is_pre_election: true
I20260925 10:12:47.282553 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.278822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.3.152.195:52981)
0925 10:12:47.279024 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.279033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.279123 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.279252 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.279253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.279254 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.279334 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.279336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.279991 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.279993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.280698 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.280701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.281490 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.281491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.281581 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.281584 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.281660 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.281701 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
I20260925 10:12:47.282668 5720 raft_consensus.cc:2454] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b216211c5764fec93f94282d6d5169f in term 0.
I20260925 10:12:47.282881 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863ce6fe628c48e68493859cd46f6eed" candidate_uuid: "3b216211c5764fec93f94282d6d5169f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a218faf652494a378fd285cc2b8b5c9c" is_pre_election: true
I20260925 10:12:47.282935 5794 leader_election.cc:302] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, 500e298a1c114938b317de53f4fb675d; no voters:
I20260925 10:12:47.283049 5579 raft_consensus.cc:2454] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b216211c5764fec93f94282d6d5169f in term 0.
I20260925 10:12:47.283073 5935 raft_consensus.cc:2781] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:47.283136 5935 raft_consensus.cc:482] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:47.283171 5935 raft_consensus.cc:3037] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.283883 5935 raft_consensus.cc:504] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.284060 5935 leader_election.cc:288] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [CANDIDATE]: Term 1 election: Requested vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053)
I20260925 10:12:47.284204 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863ce6fe628c48e68493859cd46f6eed" candidate_uuid: "3b216211c5764fec93f94282d6d5169f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a218faf652494a378fd285cc2b8b5c9c"
I20260925 10:12:47.284204 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863ce6fe628c48e68493859cd46f6eed" candidate_uuid: "3b216211c5764fec93f94282d6d5169f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d"
I20260925 10:12:47.284292 5579 raft_consensus.cc:3037] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.284292 5720 raft_consensus.cc:3037] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.285261 5720 raft_consensus.cc:2454] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b216211c5764fec93f94282d6d5169f in term 1.
I20260925 10:12:47.285254 5579 raft_consensus.cc:2454] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b216211c5764fec93f94282d6d5169f in term 1.
I20260925 10:12:47.285435 5794 leader_election.cc:302] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, 500e298a1c114938b317de53f4fb675d; no voters:
I20260925 10:12:47.285555 5935 raft_consensus.cc:2781] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:47.285797 5935 raft_consensus.cc:686] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 1 LEADER]: Becoming Leader. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Running, Role: LEADER
I20260925 10:12:47.285933 5935 consensus_queue.cc:237] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } }
I20260925 10:12:47.286831 5420 catalog_manager.cc:5495] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b216211c5764fec93f94282d6d5169f (127.3.152.195). New cstate: current_term: 1 leader_uuid: "3b216211c5764fec93f94282d6d5169f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } health_report { overall_health: UNKNOWN } } }
W20260925 10:12:47.297895 5767 tablet.cc:2369] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 10:12:47 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331166, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:47.336586 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.329014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.3.152.254:41259)
0925 10:12:47.329091 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.329094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.329275 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:12:47.329323 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.329324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.329334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.329384 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:47.329873 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.330353 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.331092 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.331271 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.331321 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.333759 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:47.333760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:47.333760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:47.333767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:47.335217 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.335662 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.335663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.335663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.335717 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:47.335950 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:47.335951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:47.335951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:47.336119 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:47.336172 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.336358 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.336479 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":44}
I20260925 10:12:47.336542 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.328979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41259 (local address 127.0.0.1:38622)
0925 10:12:47.329190 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.329201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.329275 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.329391 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.329392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.329754 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:47.329807 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.329809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.330433 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.330434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.331016 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.331018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.331335 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.331336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.331377 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.331925 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:47.331930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:47.333650 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:47.335313 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.335321 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.335323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.335528 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.335812 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:47.335813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:47.335813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:47.335888 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:47.336200 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:47.336209 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:47.336258 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.336328 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.336433 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":123,"threads_started":1}
I20260925 10:12:47.338922 5420 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:38622:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:12:47.339138 5420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:47.340797 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 20443c7a8956490193aa85d419fd14db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3222543601c041fc9af05e9fd649c0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:47.340796 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 20443c7a8956490193aa85d419fd14db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3222543601c041fc9af05e9fd649c0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:47.340807 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 20443c7a8956490193aa85d419fd14db (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3222543601c041fc9af05e9fd649c0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:12:47.340940 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20443c7a8956490193aa85d419fd14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.340961 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20443c7a8956490193aa85d419fd14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.340961 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20443c7a8956490193aa85d419fd14db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.342444 5931 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: Bootstrap starting.
I20260925 10:12:47.342612 5937 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: Bootstrap starting.
I20260925 10:12:47.342600 5932 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: Bootstrap starting.
I20260925 10:12:47.342837 5931 tablet_bootstrap.cc:654] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.343339 5937 tablet_bootstrap.cc:654] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.343395 5932 tablet_bootstrap.cc:654] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.344173 5932 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: No bootstrap required, opened a new log
I20260925 10:12:47.344240 5932 ts_tablet_manager.cc:1397] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 10:12:47.344419 5932 raft_consensus.cc:348] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.344522 5932 raft_consensus.cc:374] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.344524 5937 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: No bootstrap required, opened a new log
I20260925 10:12:47.344533 5931 tablet_bootstrap.cc:492] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: No bootstrap required, opened a new log
I20260925 10:12:47.344553 5932 raft_consensus.cc:729] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.344578 5931 ts_tablet_manager.cc:1397] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:12:47.344573 5937 ts_tablet_manager.cc:1397] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:47.344599 5932 consensus_queue.cc:260] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.344698 5932 ts_tablet_manager.cc:1428] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.344761 5931 raft_consensus.cc:348] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.344765 5937 raft_consensus.cc:348] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.344846 5937 raft_consensus.cc:374] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.344846 5931 raft_consensus.cc:374] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.344871 5931 raft_consensus.cc:729] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.344871 5937 raft_consensus.cc:729] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.344923 5931 consensus_queue.cc:260] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.344923 5937 consensus_queue.cc:260] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.345021 5931 ts_tablet_manager.cc:1428] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:47.345021 5937 ts_tablet_manager.cc:1428] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:47.345201 5936 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.345316 5936 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.345479 5935 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.345578 5935 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.345734 5939 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.345839 5939 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 10:12:47.387112 5627 tablet.cc:2369] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:12:47.463013 5908 tablet.cc:2369] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:12:47.520246 5939 raft_consensus.cc:482] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:47.520375 5939 raft_consensus.cc:504] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.520848 5939 leader_election.cc:288] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:47.525095 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.521401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:45685 (local address 127.3.152.195:44249)
0925 10:12:47.521566 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.521568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.521601 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:12:47.521626 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.521627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.521636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.521686 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.521899 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.522635 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524110 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.524657 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524744 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.524755 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.524979 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.525022 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 10:12:47.525118 5953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.521242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:44249 (local address 127.3.152.193:45685)
0925 10:12:47.521325 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.521336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.521409 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.521730 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.521731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.521744 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.521795 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.521796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.522759 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.522759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.524011 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.524014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.524793 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.524881 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.524885 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.524966 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.525014 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":23,"thread_start_us":43,"threads_started":1}
I20260925 10:12:47.525112 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.521174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42209 (local address 127.3.152.194:38669)
0925 10:12:47.521234 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.521236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.521326 (+ 90us) server_negotiation.cc:407] Connection header received
0925 10:12:47.521440 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.521441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.521452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.521484 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:47.521711 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.522612 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524093 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.524750 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524824 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.524831 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.524999 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.525049 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:47.525334 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.521061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38669 (local address 127.3.152.193:42209)
0925 10:12:47.521252 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.521265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.521357 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.521566 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.521567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.521570 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:47.521631 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.521633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.522724 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.522725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.524011 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.524014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.524820 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.524821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.524894 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.524898 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.525196 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.525254 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":30,"thread_start_us":71,"threads_started":1}
I20260925 10:12:47.525606 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20443c7a8956490193aa85d419fd14db" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d" is_pre_election: true
I20260925 10:12:47.525632 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20443c7a8956490193aa85d419fd14db" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f" is_pre_election: true
I20260925 10:12:47.525724 5720 raft_consensus.cc:2454] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:47.525741 5861 raft_consensus.cc:2454] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:47.525918 5513 leader_election.cc:302] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:47.526086 5939 raft_consensus.cc:2781] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:47.526144 5939 raft_consensus.cc:482] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:47.526176 5939 raft_consensus.cc:3037] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.526868 5939 raft_consensus.cc:504] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.527017 5939 leader_election.cc:288] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Requested vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:47.527244 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20443c7a8956490193aa85d419fd14db" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d"
I20260925 10:12:47.527336 5720 raft_consensus.cc:3037] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.527283 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20443c7a8956490193aa85d419fd14db" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f"
I20260925 10:12:47.527379 5861 raft_consensus.cc:3037] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.528277 5861 raft_consensus.cc:2454] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:47.528362 5720 raft_consensus.cc:2454] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:47.528448 5513 leader_election.cc:302] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:47.528540 5939 raft_consensus.cc:2781] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:47.528663 5939 raft_consensus.cc:686] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [term 1 LEADER]: Becoming Leader. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Running, Role: LEADER
I20260925 10:12:47.528765 5939 consensus_queue.cc:237] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.529299 5954 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: TxnStatusTablet state changed. Reason: New leader a218faf652494a378fd285cc2b8b5c9c. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.529480 5954 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:47.529696 5956 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:12:47.529811 5956 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:12:47.529737 5421 catalog_manager.cc:5495] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a218faf652494a378fd285cc2b8b5c9c (127.3.152.193). New cstate: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:47.575515 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.572412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.0.0.1:49198)
0925 10:12:47.572485 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.572498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.572566 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.572758 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.572759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.572760 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:47.572808 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.572811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.573782 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.573783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.574417 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.574419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.574898 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.574899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.575003 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.575033 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.575375 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.575383 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.575413 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.575452 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:12:47.575531 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.572497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.3.152.193:35053)
0925 10:12:47.572546 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.572548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.572559 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:47.572643 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.572644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.572666 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.572758 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.572850 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.573664 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.574504 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.574816 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.574884 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.575166 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.575305 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.575393 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.575417 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.575442 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:12:47.578315 5720 raft_consensus.cc:1261] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.578334 5861 raft_consensus.cc:1261] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.578666 5939 consensus_queue.cc:1035] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:47.578822 5954 consensus_queue.cc:1035] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:47.580039 5936 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: TxnStatusTablet state changed. Reason: New leader a218faf652494a378fd285cc2b8b5c9c. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.580152 5936 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.580524 5935 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: TxnStatusTablet state changed. Reason: New leader a218faf652494a378fd285cc2b8b5c9c. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.580605 5935 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.581138 5954 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.581233 5954 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:47.581287 5954 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.581331 5954 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P a218faf652494a378fd285cc2b8b5c9c: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:12:47.581384 5959 mvcc.cc:205] Tried to move back new op lower bound from 7333196462397718528 to 7333196462198538240. Current Snapshot: MvccSnapshot[applied={T|T < 7333196462397718528}]
I20260925 10:12:47.581571 5936 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.581653 5936 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 500e298a1c114938b317de53f4fb675d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.582304 5960 mvcc.cc:205] Tried to move back new op lower bound from 7333196462397718528 to 7333196462198538240. Current Snapshot: MvccSnapshot[applied={T|T < 7333196462397718528 or (T in {7333196462397718528})}]
I20260925 10:12:47.582433 5935 tablet_replica.cc:439] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } } }
I20260925 10:12:47.582515 5935 tablet_replica.cc:442] T 20443c7a8956490193aa85d419fd14db P 3b216211c5764fec93f94282d6d5169f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:12:47.583242 5961 mvcc.cc:205] Tried to move back new op lower bound from 7333196462397718528 to 7333196462198538240. Current Snapshot: MvccSnapshot[applied={T|T < 7333196462397718528}]
I20260925 10:12:47.586129 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.583304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.3.152.195:44249)
0925 10:12:47.583343 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.583345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.583462 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:12:47.583490 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.583491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.583506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.583572 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.583730 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.584503 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.585266 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.585570 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.585641 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.585749 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.585865 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.585940 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.586026 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.586048 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:12:47.591100 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.588729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39699 (local address 127.3.152.193:35053)
0925 10:12:47.588797 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.588799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.588810 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:47.588890 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.588891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.588901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.588964 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.589137 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.589601 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.590362 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.590561 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.590644 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.590841 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.590881 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.590941 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.591002 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.591023 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:12:47.591105 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.588655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.3.152.195:39699)
0925 10:12:47.588732 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.588744 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.588826 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.588978 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.588979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.588980 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:47.589040 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.589043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.589687 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.589688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.590275 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.590277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.590647 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.590648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.590718 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.590731 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.590942 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.590944 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.590997 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.591041 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:12:47.593572 5579 raft_consensus.cc:1261] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Refusing update from remote peer 3b216211c5764fec93f94282d6d5169f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.593847 5720 raft_consensus.cc:1261] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Refusing update from remote peer 3b216211c5764fec93f94282d6d5169f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.593881 5935 consensus_queue.cc:1035] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:47.594061 5941 consensus_queue.cc:1035] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [LEADER]: Connected to new peer: Peer: permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:47.603250 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.600705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:45157 (local address 127.3.152.195:44249)
0925 10:12:47.600760 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.600763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.600871 (+ 108us) server_negotiation.cc:407] Connection header received
0925 10:12:47.601007 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.601008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.601021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.601070 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.601214 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.601686 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.602466 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.602755 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.602815 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.603009 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.603044 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.603091 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.603149 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.603167 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:47.603264 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.600498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:44249 (local address 127.3.152.193:45157)
0925 10:12:47.600792 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.600805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.600900 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.601087 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.601088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.601090 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:47.601141 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.601143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.601774 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.601775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.602364 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.602366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.602849 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.602850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.602915 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.602931 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.603092 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.603094 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.603145 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.603189 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":90,"threads_started":1}
I20260925 10:12:47.610390 5420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:47.610522 5420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:47.611222 5420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Sending DeleteTablet for 3 replicas of tablet 863ce6fe628c48e68493859cd46f6eed
I20260925 10:12:47.611611 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 863ce6fe628c48e68493859cd46f6eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:47 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49182
I20260925 10:12:47.611618 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 863ce6fe628c48e68493859cd46f6eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:47 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39562
I20260925 10:12:47.611857 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet 863ce6fe628c48e68493859cd46f6eed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:47 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:46528
I20260925 10:12:47.611989 5979 tablet_replica.cc:330] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: stopping tablet replica
I20260925 10:12:47.612130 5979 raft_consensus.cc:2229] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:47.612247 5979 raft_consensus.cc:2258] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:47.612407 5420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:47.612568 5420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:47.612977 5979 ts_tablet_manager.cc:1905] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:47.613759 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 (DEFAULT_TABLE table=test-table [id=6af698189f4941f69ca91811c1aab231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.613878 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b5f8e16aaa473f9b4317cccd570301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.614888 5979 ts_tablet_manager.cc:1918] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:47.614950 5979 log.cc:1196] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/863ce6fe628c48e68493859cd46f6eed
I20260925 10:12:47.615177 5937 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Bootstrap starting.
I20260925 10:12:47.615355 5979 ts_tablet_manager.cc:1939] T 863ce6fe628c48e68493859cd46f6eed P a218faf652494a378fd285cc2b8b5c9c: Deleting consensus metadata
I20260925 10:12:47.615597 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 (DEFAULT_TABLE table=test-table [id=6af698189f4941f69ca91811c1aab231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.615708 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b5f8e16aaa473f9b4317cccd570301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.615772 5937 tablet_bootstrap.cc:654] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.615767 5407 catalog_manager.cc:4841] TS a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053): tablet 863ce6fe628c48e68493859cd46f6eed (table test-table [id=d7e112f0008f4771aefddee62fa5008f]) successfully deleted
I20260925 10:12:47.616101 5978 tablet_replica.cc:330] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: stopping tablet replica
I20260925 10:12:47.616200 5978 raft_consensus.cc:2229] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:47.616348 5978 raft_consensus.cc:2258] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:47.616792 5978 ts_tablet_manager.cc:1905] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:47.617157 5932 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Bootstrap starting.
I20260925 10:12:47.617341 5841 tablet_service.cc:1467] Processing CreateTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 (DEFAULT_TABLE table=test-table [id=6af698189f4941f69ca91811c1aab231]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:47.617451 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b5f8e16aaa473f9b4317cccd570301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:47.617570 5932 tablet_bootstrap.cc:654] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.618846 5931 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Bootstrap starting.
I20260925 10:12:47.619269 5978 ts_tablet_manager.cc:1918] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:47.619297 5931 tablet_bootstrap.cc:654] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:47.619340 5978 log.cc:1196] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/863ce6fe628c48e68493859cd46f6eed
I20260925 10:12:47.619379 5980 tablet_replica.cc:330] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: stopping tablet replica
I20260925 10:12:47.619478 5980 raft_consensus.cc:2229] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:47.619592 5978 ts_tablet_manager.cc:1939] T 863ce6fe628c48e68493859cd46f6eed P 500e298a1c114938b317de53f4fb675d: Deleting consensus metadata
I20260925 10:12:47.619590 5980 raft_consensus.cc:2258] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:47.620000 5408 catalog_manager.cc:4841] TS 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669): tablet 863ce6fe628c48e68493859cd46f6eed (table test-table [id=d7e112f0008f4771aefddee62fa5008f]) successfully deleted
I20260925 10:12:47.620028 5980 ts_tablet_manager.cc:1905] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:47.620103 5937 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: No bootstrap required, opened a new log
I20260925 10:12:47.620157 5937 ts_tablet_manager.cc:1397] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 10:12:47.620297 5937 raft_consensus.cc:348] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.620368 5937 raft_consensus.cc:374] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.620388 5937 raft_consensus.cc:729] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.620421 5932 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: No bootstrap required, opened a new log
I20260925 10:12:47.620466 5932 ts_tablet_manager.cc:1397] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260925 10:12:47.620431 5937 consensus_queue.cc:260] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.620537 5937 ts_tablet_manager.cc:1428] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.620590 5932 raft_consensus.cc:348] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.620656 5932 raft_consensus.cc:374] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.620679 5932 raft_consensus.cc:729] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.620723 5932 consensus_queue.cc:260] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.620808 5932 ts_tablet_manager.cc:1428] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:47.621889 5980 ts_tablet_manager.cc:1918] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:47.621979 5980 log.cc:1196] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/863ce6fe628c48e68493859cd46f6eed
I20260925 10:12:47.622212 5980 ts_tablet_manager.cc:1939] T 863ce6fe628c48e68493859cd46f6eed P 3b216211c5764fec93f94282d6d5169f: Deleting consensus metadata
I20260925 10:12:47.622581 5408 catalog_manager.cc:4841] TS 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249): tablet 863ce6fe628c48e68493859cd46f6eed (table test-table [id=d7e112f0008f4771aefddee62fa5008f]) successfully deleted
I20260925 10:12:47.624354 5931 tablet_bootstrap.cc:492] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: No bootstrap required, opened a new log
I20260925 10:12:47.624423 5931 ts_tablet_manager.cc:1397] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260925 10:12:47.624569 5931 raft_consensus.cc:348] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.624644 5931 raft_consensus.cc:374] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:47.624672 5931 raft_consensus.cc:729] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Initialized, Role: FOLLOWER
I20260925 10:12:47.624713 5931 consensus_queue.cc:260] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.624806 5931 ts_tablet_manager.cc:1428] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:47.639567 5954 raft_consensus.cc:482] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:47.639688 5954 raft_consensus.cc:504] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.639844 5954 leader_election.cc:288] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:47.640117 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b5f8e16aaa473f9b4317cccd570301" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d" is_pre_election: true
I20260925 10:12:47.640103 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b5f8e16aaa473f9b4317cccd570301" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f" is_pre_election: true
I20260925 10:12:47.640199 5861 raft_consensus.cc:2454] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:47.640199 5719 raft_consensus.cc:2454] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:47.640363 5513 leader_election.cc:302] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500e298a1c114938b317de53f4fb675d, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:47.640453 5954 raft_consensus.cc:2781] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:47.640487 5954 raft_consensus.cc:482] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:47.640511 5954 raft_consensus.cc:3037] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.641297 5954 raft_consensus.cc:504] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.641440 5954 leader_election.cc:288] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Requested vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:47.641644 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b5f8e16aaa473f9b4317cccd570301" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d"
I20260925 10:12:47.641659 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b5f8e16aaa473f9b4317cccd570301" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f"
I20260925 10:12:47.641722 5861 raft_consensus.cc:3037] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.641722 5719 raft_consensus.cc:3037] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:47.642542 5719 raft_consensus.cc:2454] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:47.642542 5861 raft_consensus.cc:2454] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:47.642704 5513 leader_election.cc:302] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:47.642915 5954 raft_consensus.cc:2781] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:47.642967 5954 raft_consensus.cc:686] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 1 LEADER]: Becoming Leader. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Running, Role: LEADER
I20260925 10:12:47.643021 5954 consensus_queue.cc:237] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:47.643627 5419 catalog_manager.cc:5495] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a218faf652494a378fd285cc2b8b5c9c (127.3.152.193). New cstate: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:47.655472 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.653141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.3.152.193:35053)
0925 10:12:47.653183 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.653185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.653240 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:12:47.653326 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.653327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.653340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.653389 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.653528 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.654005 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.654737 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.654950 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.655035 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.655189 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.655224 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.655297 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.655359 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.655380 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:12:47.660020 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.657663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.3.152.193:58675)
0925 10:12:47.657713 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:47.657724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:47.657763 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:47.657876 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:47.657876 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:47.657878 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:47.657924 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.657925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.658574 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.658575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.659192 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:47.659194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.659591 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.659592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:47.659638 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.659644 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.659816 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.659818 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:47.659857 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 10:12:47.659900 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":3}
I20260925 10:12:47.660020 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:47.657699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:58675 (local address 127.3.152.193:35053)
0925 10:12:47.657742 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:47.657744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:47.657751 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:12:47.657819 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:47.657820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:47.657829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:47.657876 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:47.657979 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.658499 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.659289 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:47.659510 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:47.659582 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:47.659727 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:47.659762 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:47.659809 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:47.659884 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 10:12:47.659898 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":21}
I20260925 10:12:47.662492 5719 raft_consensus.cc:1261] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.662485 5861 raft_consensus.cc:1261] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:47.662730 5939 consensus_queue.cc:1035] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:47.662858 5939 consensus_queue.cc:1035] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:12:47.668821 5959 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:12:49.193817 5420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:49.193928 5420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:49.194869 5420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Sending DeleteTablet for 3 replicas of tablet 75b5f8e16aaa473f9b4317cccd570301
I20260925 10:12:49.195138 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49182
I20260925 10:12:49.195230 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:46528
I20260925 10:12:49.195361 6008 tablet_replica.cc:330] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: stopping tablet replica
I20260925 10:12:49.195422 6008 raft_consensus.cc:2229] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:49.195523 6008 raft_consensus.cc:2258] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.195822 6008 ts_tablet_manager.cc:1905] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.196095 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b5f8e16aaa473f9b4317cccd570301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39562
I20260925 10:12:49.197084 6010 tablet_replica.cc:330] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: stopping tablet replica
I20260925 10:12:49.197209 6010 raft_consensus.cc:2229] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:49.197338 6010 raft_consensus.cc:2258] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.197666 6010 ts_tablet_manager.cc:1905] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.197868 6008 ts_tablet_manager.cc:1918] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:12:49.197976 6008 log.cc:1196] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/75b5f8e16aaa473f9b4317cccd570301
I20260925 10:12:49.198246 6009 tablet_replica.cc:330] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: stopping tablet replica
I20260925 10:12:49.198319 6009 raft_consensus.cc:2229] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:49.198321 6008 ts_tablet_manager.cc:1939] T 75b5f8e16aaa473f9b4317cccd570301 P a218faf652494a378fd285cc2b8b5c9c: Deleting consensus metadata
I20260925 10:12:49.198406 6009 raft_consensus.cc:2258] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.198661 5407 catalog_manager.cc:4841] TS a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053): tablet 75b5f8e16aaa473f9b4317cccd570301 (table test-table [id=6af698189f4941f69ca91811c1aab231]) successfully deleted
I20260925 10:12:49.198886 6009 ts_tablet_manager.cc:1905] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.199376 6010 ts_tablet_manager.cc:1918] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:12:49.199440 6010 log.cc:1196] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/75b5f8e16aaa473f9b4317cccd570301
I20260925 10:12:49.199651 6010 ts_tablet_manager.cc:1939] T 75b5f8e16aaa473f9b4317cccd570301 P 500e298a1c114938b317de53f4fb675d: Deleting consensus metadata
I20260925 10:12:49.200170 5408 catalog_manager.cc:4841] TS 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669): tablet 75b5f8e16aaa473f9b4317cccd570301 (table test-table [id=6af698189f4941f69ca91811c1aab231]) successfully deleted
I20260925 10:12:49.200726 6009 ts_tablet_manager.cc:1918] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:12:49.200791 6009 log.cc:1196] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/75b5f8e16aaa473f9b4317cccd570301
I20260925 10:12:49.200996 6009 ts_tablet_manager.cc:1939] T 75b5f8e16aaa473f9b4317cccd570301 P 3b216211c5764fec93f94282d6d5169f: Deleting consensus metadata
I20260925 10:12:49.201538 5408 catalog_manager.cc:4841] TS 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249): tablet 75b5f8e16aaa473f9b4317cccd570301 (table test-table [id=6af698189f4941f69ca91811c1aab231]) successfully deleted
Sep 25 10:12:49 dist-test-slave-dzwq krb5kdc[5376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:12:49 dist-test-slave-dzwq krb5kdc[5376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331169, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:49.225785 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.214000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.3.152.254:41259)
0925 10:12:49.214175 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.214178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.214205 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:12:49.214303 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.214304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.214315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.214400 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:49.215111 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.215662 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.216611 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.216875 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.216967 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.223190 (+ 6223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:49.223192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:49.223192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:49.223201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:49.224660 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:49.224999 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:49.225000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:49.225000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:49.225054 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:49.225276 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:49.225277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:49.225277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:49.225393 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:49.225473 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.225636 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.225705 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260925 10:12:49.227365 5420 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:12:49.227519 5420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:49.229111 5559 tablet_service.cc:1467] Processing CreateTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a (DEFAULT_TABLE table=test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.229137 5700 tablet_service.cc:1467] Processing CreateTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a (DEFAULT_TABLE table=test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.229137 5841 tablet_service.cc:1467] Processing CreateTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a (DEFAULT_TABLE table=test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.229265 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecec4a132d1041f89aa645d89c5c8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.229266 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecec4a132d1041f89aa645d89c5c8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.229314 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecec4a132d1041f89aa645d89c5c8c7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.230521 6019 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Bootstrap starting.
I20260925 10:12:49.230701 6021 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Bootstrap starting.
I20260925 10:12:49.230945 6019 tablet_bootstrap.cc:654] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.231036 6021 tablet_bootstrap.cc:654] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.231379 6020 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Bootstrap starting.
I20260925 10:12:49.231554 6019 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: No bootstrap required, opened a new log
I20260925 10:12:49.231633 6019 ts_tablet_manager.cc:1397] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:49.231782 6020 tablet_bootstrap.cc:654] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.231769 6019 raft_consensus.cc:348] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.231875 6019 raft_consensus.cc:374] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.231906 6019 raft_consensus.cc:729] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.232005 6019 consensus_queue.cc:260] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.232224 6019 ts_tablet_manager.cc:1428] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:49.232293 6021 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: No bootstrap required, opened a new log
I20260925 10:12:49.232355 6021 ts_tablet_manager.cc:1397] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:12:49.232668 6021 raft_consensus.cc:348] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.232798 6021 raft_consensus.cc:374] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.232841 6021 raft_consensus.cc:729] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.232932 6021 consensus_queue.cc:260] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.232999 6020 tablet_bootstrap.cc:492] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: No bootstrap required, opened a new log
I20260925 10:12:49.233088 6020 ts_tablet_manager.cc:1397] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:12:49.233127 6021 ts_tablet_manager.cc:1428] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:49.233251 6020 raft_consensus.cc:348] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.233349 6020 raft_consensus.cc:374] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.233374 6020 raft_consensus.cc:729] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.233424 6020 consensus_queue.cc:260] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.233637 6020 ts_tablet_manager.cc:1428] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:49.316458 6026 raft_consensus.cc:482] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:49.316624 6026 raft_consensus.cc:504] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.317000 6026 leader_election.cc:288] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:49.320293 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.317273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42221 (local address 127.3.152.195:44249)
0925 10:12:49.317456 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.317458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.317469 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:49.317542 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.317542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.317562 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.317645 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:49.317857 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.318384 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319470 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.319870 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319961 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.319970 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.320165 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.320209 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260925 10:12:49.320294 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.317284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:48623 (local address 127.3.152.193:35053)
0925 10:12:49.317529 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.317531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.317541 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:49.317620 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.317621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.317631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.317720 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:12:49.317887 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.318379 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319485 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.319880 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319971 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.319996 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.320164 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.320211 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260925 10:12:49.320325 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.317218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.3.152.194:48623)
0925 10:12:49.317443 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:49.317488 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:49.317551 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:49.317724 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:49.317725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:49.317727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:49.317801 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.317803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.318471 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.318472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.319369 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.319371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.319992 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.320074 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.320081 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:12:49.320146 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:12:49.320205 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":25,"thread_start_us":133,"threads_started":1}
I20260925 10:12:49.320328 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.317186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:44249 (local address 127.3.152.194:42221)
0925 10:12:49.317399 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:49.317414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:49.317479 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:49.317665 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:49.317666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:49.317668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:12:49.317749 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.317751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.318474 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.318475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.319370 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.319372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.319984 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.319986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.320074 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.320079 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:12:49.320147 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:12:49.320205 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
I20260925 10:12:49.320600 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecec4a132d1041f89aa645d89c5c8c7a" candidate_uuid: "500e298a1c114938b317de53f4fb675d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a218faf652494a378fd285cc2b8b5c9c" is_pre_election: true
I20260925 10:12:49.320693 5579 raft_consensus.cc:2454] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500e298a1c114938b317de53f4fb675d in term 0.
I20260925 10:12:49.320850 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecec4a132d1041f89aa645d89c5c8c7a" candidate_uuid: "500e298a1c114938b317de53f4fb675d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f" is_pre_election: true
I20260925 10:12:49.320874 5653 leader_election.cc:302] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500e298a1c114938b317de53f4fb675d, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:49.320945 5861 raft_consensus.cc:2454] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 500e298a1c114938b317de53f4fb675d in term 0.
I20260925 10:12:49.321009 6026 raft_consensus.cc:2781] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:49.321051 6026 raft_consensus.cc:482] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:49.321077 6026 raft_consensus.cc:3037] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.321800 6026 raft_consensus.cc:504] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.321913 6026 leader_election.cc:288] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [CANDIDATE]: Term 1 election: Requested vote from peers a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:49.322088 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecec4a132d1041f89aa645d89c5c8c7a" candidate_uuid: "500e298a1c114938b317de53f4fb675d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f"
I20260925 10:12:49.322085 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecec4a132d1041f89aa645d89c5c8c7a" candidate_uuid: "500e298a1c114938b317de53f4fb675d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a218faf652494a378fd285cc2b8b5c9c"
I20260925 10:12:49.322161 5861 raft_consensus.cc:3037] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.322181 5579 raft_consensus.cc:3037] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.322896 5861 raft_consensus.cc:2454] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500e298a1c114938b317de53f4fb675d in term 1.
I20260925 10:12:49.322903 5579 raft_consensus.cc:2454] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 500e298a1c114938b317de53f4fb675d in term 1.
I20260925 10:12:49.323079 5654 leader_election.cc:302] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b216211c5764fec93f94282d6d5169f, 500e298a1c114938b317de53f4fb675d; no voters:
I20260925 10:12:49.323187 6026 raft_consensus.cc:2781] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:49.323339 6026 raft_consensus.cc:686] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 1 LEADER]: Becoming Leader. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Running, Role: LEADER
I20260925 10:12:49.323421 6026 consensus_queue.cc:237] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.324246 5420 catalog_manager.cc:5495] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d reported cstate change: term changed from 0 to 1, leader changed from <none> to 500e298a1c114938b317de53f4fb675d (127.3.152.194). New cstate: current_term: 1 leader_uuid: "500e298a1c114938b317de53f4fb675d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:49.353945 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.351207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.3.152.194:38669)
0925 10:12:49.351393 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.351395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.351401 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:12:49.351435 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.351436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.351455 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.351544 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:49.351701 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.352259 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.353055 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.353299 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.353392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.353562 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.353683 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.353771 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.353811 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.353835 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260925 10:12:49.359280 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.356909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:43795 (local address 127.3.152.193:35053)
0925 10:12:49.356964 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.356966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.356977 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:49.357003 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.357004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.357012 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.357061 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:49.357196 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.357692 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.358489 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.358718 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.358772 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.358978 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.359034 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.359114 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.359197 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.359218 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:12:49.359295 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.356594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.3.152.194:43795)
0925 10:12:49.356850 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:49.356861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:49.356908 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:49.357078 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:49.357079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:49.357081 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:49.357138 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.357140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.357787 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.357788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.358409 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.358412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.358809 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.358810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.358875 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.358890 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.359124 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.359131 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:12:49.359179 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:12:49.359223 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":162,"threads_started":1}
I20260925 10:12:49.361753 5861 raft_consensus.cc:1261] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Refusing update from remote peer 500e298a1c114938b317de53f4fb675d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:49.361757 5579 raft_consensus.cc:1261] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Refusing update from remote peer 500e298a1c114938b317de53f4fb675d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:49.362042 6031 consensus_queue.cc:1035] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:49.362252 6026 consensus_queue.cc:1035] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:49.369899 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.367204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:60425 (local address 127.3.152.194:38669)
0925 10:12:49.367250 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.367252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.367393 (+ 141us) server_negotiation.cc:407] Connection header received
0925 10:12:49.367425 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.367426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.367437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.367497 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:49.367647 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.368257 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.369061 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.369315 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.369394 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.369610 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.369651 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.369707 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.369795 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.369818 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:49.369904 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.367172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38669 (local address 127.3.152.193:60425)
0925 10:12:49.367309 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:49.367321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:49.367383 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:49.367500 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:49.367501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:49.367502 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:12:49.367546 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.367569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.368337 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.368338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.368948 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:49.368950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.369442 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.369443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:49.369526 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.369532 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.369711 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.369713 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:12:49.369769 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:12:49.369823 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":94,"threads_started":1}
I20260925 10:12:49.376636 5419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:49.376765 5419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:49.377468 5419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Sending DeleteTablet for 3 replicas of tablet ecec4a132d1041f89aa645d89c5c8c7a
I20260925 10:12:49.377846 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49182
I20260925 10:12:49.377851 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39562
I20260925 10:12:49.377874 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet ecec4a132d1041f89aa645d89c5c8c7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:46528
I20260925 10:12:49.377951 6010 tablet_replica.cc:330] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: stopping tablet replica
I20260925 10:12:49.377964 6008 tablet_replica.cc:330] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: stopping tablet replica
I20260925 10:12:49.377996 6010 raft_consensus.cc:2229] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:12:49.378015 6008 raft_consensus.cc:2229] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:49.378048 6008 raft_consensus.cc:2258] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.378087 6010 raft_consensus.cc:2258] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.378124 6009 tablet_replica.cc:330] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: stopping tablet replica
I20260925 10:12:49.378183 6009 raft_consensus.cc:2229] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:49.378226 6009 raft_consensus.cc:2258] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:49.378487 6010 ts_tablet_manager.cc:1905] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.378611 6009 ts_tablet_manager.cc:1905] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.378734 6008 ts_tablet_manager.cc:1905] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:49.380388 6010 ts_tablet_manager.cc:1918] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:49.380314 5419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:12:49.380429 6009 ts_tablet_manager.cc:1918] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:49.380450 6010 log.cc:1196] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/ecec4a132d1041f89aa645d89c5c8c7a
I20260925 10:12:49.380474 6009 log.cc:1196] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/ecec4a132d1041f89aa645d89c5c8c7a
W20260925 10:12:49.380481 5419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:12:49.380678 6010 ts_tablet_manager.cc:1939] T ecec4a132d1041f89aa645d89c5c8c7a P 500e298a1c114938b317de53f4fb675d: Deleting consensus metadata
I20260925 10:12:49.380743 6009 ts_tablet_manager.cc:1939] T ecec4a132d1041f89aa645d89c5c8c7a P 3b216211c5764fec93f94282d6d5169f: Deleting consensus metadata
I20260925 10:12:49.381043 5408 catalog_manager.cc:4841] TS 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669): tablet ecec4a132d1041f89aa645d89c5c8c7a (table test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]) successfully deleted
I20260925 10:12:49.381136 5408 catalog_manager.cc:4841] TS 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249): tablet ecec4a132d1041f89aa645d89c5c8c7a (table test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]) successfully deleted
I20260925 10:12:49.381127 6008 ts_tablet_manager.cc:1918] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:12:49.381176 6008 log.cc:1196] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/ecec4a132d1041f89aa645d89c5c8c7a
I20260925 10:12:49.381389 6008 ts_tablet_manager.cc:1939] T ecec4a132d1041f89aa645d89c5c8c7a P a218faf652494a378fd285cc2b8b5c9c: Deleting consensus metadata
I20260925 10:12:49.381752 5407 catalog_manager.cc:4841] TS a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053): tablet ecec4a132d1041f89aa645d89c5c8c7a (table test-table [id=6c5aba1f366a4e4dae4611b8d8a8f08b]) successfully deleted
I20260925 10:12:49.381935 5559 tablet_service.cc:1467] Processing CreateTablet for tablet f1a3403f298d45f48e05c66f11daa642 (DEFAULT_TABLE table=test-table [id=00a1a9e8138847f3baa016994719a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.381963 5841 tablet_service.cc:1467] Processing CreateTablet for tablet f1a3403f298d45f48e05c66f11daa642 (DEFAULT_TABLE table=test-table [id=00a1a9e8138847f3baa016994719a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.382050 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a3403f298d45f48e05c66f11daa642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.382028 5700 tablet_service.cc:1467] Processing CreateTablet for tablet f1a3403f298d45f48e05c66f11daa642 (DEFAULT_TABLE table=test-table [id=00a1a9e8138847f3baa016994719a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:12:49.382067 5841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a3403f298d45f48e05c66f11daa642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.382133 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a3403f298d45f48e05c66f11daa642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:49.383292 6020 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c: Bootstrap starting.
I20260925 10:12:49.383309 6019 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Bootstrap starting.
I20260925 10:12:49.383360 6021 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: Bootstrap starting.
I20260925 10:12:49.383625 6020 tablet_bootstrap.cc:654] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.383639 6019 tablet_bootstrap.cc:654] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.383765 6021 tablet_bootstrap.cc:654] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:49.384222 6019 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: No bootstrap required, opened a new log
I20260925 10:12:49.384274 6019 ts_tablet_manager.cc:1397] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:49.384368 6020 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c: No bootstrap required, opened a new log
I20260925 10:12:49.384411 6020 ts_tablet_manager.cc:1397] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:49.384387 6019 raft_consensus.cc:348] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.384455 6019 raft_consensus.cc:374] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.384477 6019 raft_consensus.cc:729] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b216211c5764fec93f94282d6d5169f, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.384519 6019 consensus_queue.cc:260] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.384521 6020 raft_consensus.cc:348] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.384580 6020 raft_consensus.cc:374] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.384600 6020 raft_consensus.cc:729] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.384622 6019 ts_tablet_manager.cc:1428] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:49.384639 6020 consensus_queue.cc:260] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.384734 6020 ts_tablet_manager.cc:1428] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:49.384800 6021 tablet_bootstrap.cc:492] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: No bootstrap required, opened a new log
I20260925 10:12:49.384847 6021 ts_tablet_manager.cc:1397] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:49.385018 6021 raft_consensus.cc:348] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.385089 6021 raft_consensus.cc:374] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:49.385115 6021 raft_consensus.cc:729] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e298a1c114938b317de53f4fb675d, State: Initialized, Role: FOLLOWER
I20260925 10:12:49.385152 6021 consensus_queue.cc:260] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.385255 6021 ts_tablet_manager.cc:1428] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:49.524744 5996 raft_consensus.cc:482] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:12:49.524887 5996 raft_consensus.cc:504] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.525101 5996 leader_election.cc:288] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:49.525394 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a3403f298d45f48e05c66f11daa642" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d" is_pre_election: true
I20260925 10:12:49.525499 5719 raft_consensus.cc:2454] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:49.525445 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a3403f298d45f48e05c66f11daa642" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f" is_pre_election: true
I20260925 10:12:49.525678 5861 raft_consensus.cc:2454] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 0.
I20260925 10:12:49.525731 5513 leader_election.cc:302] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500e298a1c114938b317de53f4fb675d, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:49.525838 5996 raft_consensus.cc:2781] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:12:49.525882 5996 raft_consensus.cc:482] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:12:49.525903 5996 raft_consensus.cc:3037] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.526607 5996 raft_consensus.cc:504] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.526754 5996 leader_election.cc:288] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Requested vote from peers 500e298a1c114938b317de53f4fb675d (127.3.152.194:38669), 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249)
I20260925 10:12:49.526942 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a3403f298d45f48e05c66f11daa642" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500e298a1c114938b317de53f4fb675d"
I20260925 10:12:49.526957 5861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a3403f298d45f48e05c66f11daa642" candidate_uuid: "a218faf652494a378fd285cc2b8b5c9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b216211c5764fec93f94282d6d5169f"
I20260925 10:12:49.527017 5719 raft_consensus.cc:3037] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.527042 5861 raft_consensus.cc:3037] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:49.527808 5719 raft_consensus.cc:2454] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:49.527899 5861 raft_consensus.cc:2454] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a218faf652494a378fd285cc2b8b5c9c in term 1.
I20260925 10:12:49.528075 5513 leader_election.cc:302] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 500e298a1c114938b317de53f4fb675d, a218faf652494a378fd285cc2b8b5c9c; no voters:
I20260925 10:12:49.528169 5996 raft_consensus.cc:2781] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:49.528231 5996 raft_consensus.cc:686] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [term 1 LEADER]: Becoming Leader. State: Replica: a218faf652494a378fd285cc2b8b5c9c, State: Running, Role: LEADER
I20260925 10:12:49.528298 5996 consensus_queue.cc:237] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } }
I20260925 10:12:49.529000 5420 catalog_manager.cc:5495] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a218faf652494a378fd285cc2b8b5c9c (127.3.152.193). New cstate: current_term: 1 leader_uuid: "a218faf652494a378fd285cc2b8b5c9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a218faf652494a378fd285cc2b8b5c9c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 } health_report { overall_health: UNKNOWN } } }
I20260925 10:12:49.537847 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:49.535418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.3.152.193:35053)
0925 10:12:49.535465 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:49.535467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:49.535492 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:12:49.535565 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:49.535566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:49.535577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:49.535618 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:12:49.535715 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.536258 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.537086 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:49.537301 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:49.537357 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:49.537515 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:12:49.537542 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:12:49.537576 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:49.537709 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:12:49.537728 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:12:49.540875 5861 raft_consensus.cc:1261] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:49.540925 5719 raft_consensus.cc:1261] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Refusing update from remote peer a218faf652494a378fd285cc2b8b5c9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:12:49.541167 5939 consensus_queue.cc:1035] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b216211c5764fec93f94282d6d5169f" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:12:49.541337 5939 consensus_queue.cc:1035] T f1a3403f298d45f48e05c66f11daa642 P a218faf652494a378fd285cc2b8b5c9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "500e298a1c114938b317de53f4fb675d" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:12:49.545890 5989 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 10:12:51.063499 5419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:51.063613 5419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:12:51.064520 5419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717b7fb79dbe4fc1b1d76bfcab55c4d3: Sending DeleteTablet for 3 replicas of tablet f1a3403f298d45f48e05c66f11daa642
I20260925 10:12:51.064779 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a3403f298d45f48e05c66f11daa642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:51 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49182
I20260925 10:12:51.064826 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a3403f298d45f48e05c66f11daa642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:51 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39562
I20260925 10:12:51.064950 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5494
I20260925 10:12:51.064976 5841 tablet_service.cc:1514] Processing DeleteTablet for tablet f1a3403f298d45f48e05c66f11daa642 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:12:51 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:46528
I20260925 10:12:51.065701 6072 tablet_replica.cc:330] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: stopping tablet replica
I20260925 10:12:51.065763 6072 raft_consensus.cc:2229] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:12:51.065805 6072 raft_consensus.cc:2258] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:51.066048 6072 ts_tablet_manager.cc:1905] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:51.067939 6072 ts_tablet_manager.cc:1918] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 10:12:51.068076 6072 log.cc:1196] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/f1a3403f298d45f48e05c66f11daa642
I20260925 10:12:51.068298 6072 ts_tablet_manager.cc:1939] T f1a3403f298d45f48e05c66f11daa642 P 3b216211c5764fec93f94282d6d5169f: Deleting consensus metadata
I20260925 10:12:51.068632 5408 catalog_manager.cc:4841] TS 3b216211c5764fec93f94282d6d5169f (127.3.152.195:44249): tablet f1a3403f298d45f48e05c66f11daa642 (table test-table [id=00a1a9e8138847f3baa016994719a906]) successfully deleted
I20260925 10:12:51.072001 6071 tablet_replica.cc:330] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: stopping tablet replica
I20260925 10:12:51.072175 6071 raft_consensus.cc:2229] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 10:12:51.072152 5407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:35053 (error 108)
I20260925 10:12:51.072237 6071 raft_consensus.cc:2258] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:12:51.072486 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5635
I20260925 10:12:51.072541 6071 ts_tablet_manager.cc:1905] T f1a3403f298d45f48e05c66f11daa642 P 500e298a1c114938b317de53f4fb675d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:12:51.074411 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:51.072578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35053 (local address 127.0.0.1:49212)
0925 10:12:51.074278 (+ 1700us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:51.074326 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":67,"threads_started":1}
W20260925 10:12:51.074532 5407 catalog_manager.cc:4584] TS a218faf652494a378fd285cc2b8b5c9c (127.3.152.193:35053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1a3403f298d45f48e05c66f11daa642: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35053: connect: Connection refused (error 111)
I20260925 10:12:51.078388 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5775
I20260925 10:12:51.078423 6074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:51.078304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38669 (local address 127.0.0.1:39592)
0925 10:12:51.078359 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:51.078385 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.194:38669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 10:12:51.083473 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 5389
2026-09-25T10:12:51Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5799 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:51 dist-test-slave-dzwq krb5kdc[6078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:51 dist-test-slave-dzwq krb5kdc[6078](info): set up 2 sockets
Sep 25 10:12:51 dist-test-slave-dzwq krb5kdc[6078](info): commencing operation
krb5kdc: starting...
W20260925 10:12:52.120465 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:52.244161 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38239
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:52.326926 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:52.327153 6091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:52.327176 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:52.328349 6091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:52.328389 6091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:52.328404 6091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:52.328418 6091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:52.328429 6091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:52.329881 6091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:52.330093 6091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:52.330287 6091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:52.335331 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.335348 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.335341 6099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.335592 6091 server_base.cc:1034] running on GCE node
I20260925 10:12:52.336019 6091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:52.336383 6091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:52.337530 6091 hybrid_clock.cc:648] HybridClock initialized: now 1790331172337516 us; error 34 us; skew 500 ppm
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:52.340165 6091 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:52.340669 6091 webserver.cc:460] Webserver started at http://127.3.152.254:41361/ using document root <none> and password file <none>
I20260925 10:12:52.340924 6091 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:52.340994 6091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:52.341143 6091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:52.342219 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "d5a8a434c22e45ca8faa53cb495af025"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "49a70c5ba7bded79f936cddf92096168"
server_key_iv: "5c5ab4573f9f42a22e68b04e1a18176f"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.342604 6091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "d5a8a434c22e45ca8faa53cb495af025"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "49a70c5ba7bded79f936cddf92096168"
server_key_iv: "5c5ab4573f9f42a22e68b04e1a18176f"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.343957 6091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:12:52.344748 6106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.344931 6091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:52.345031 6091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "d5a8a434c22e45ca8faa53cb495af025"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "49a70c5ba7bded79f936cddf92096168"
server_key_iv: "5c5ab4573f9f42a22e68b04e1a18176f"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.345114 6091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:52.368342 6091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:52.370682 6091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:52.370893 6091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:52.375053 6091 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:38239
I20260925 10:12:52.375097 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:38239 every 8 connection(s)
I20260925 10:12:52.375499 6091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:52.376551 6159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:52.378994 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025: Bootstrap starting.
I20260925 10:12:52.379570 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6091
I20260925 10:12:52.379668 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:52.379727 6159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:52.379856 3683 external_mini_cluster.cc:1428] Setting key 638d26718d97c753d31ce7f5b8234b42
I20260925 10:12:52.380110 6159 log.cc:824] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:52.380908 6159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025: No bootstrap required, opened a new log
I20260925 10:12:52.382513 6159 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } }
I20260925 10:12:52.382690 6159 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:52.382750 6159 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a8a434c22e45ca8faa53cb495af025, State: Initialized, Role: FOLLOWER
I20260925 10:12:52.382913 6159 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } }
I20260925 10:12:52.382992 6159 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:52.383014 6159 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:52.383068 6159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:52.383885 6159 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } }
I20260925 10:12:52.384019 6159 leader_election.cc:302] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a8a434c22e45ca8faa53cb495af025; no voters:
I20260925 10:12:52.384222 6159 leader_election.cc:288] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:52.384248 6164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:52.384363 6164 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [term 1 LEADER]: Becoming Leader. State: Replica: d5a8a434c22e45ca8faa53cb495af025, State: Running, Role: LEADER
I20260925 10:12:52.384512 6159 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:52.384500 6164 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } }
I20260925 10:12:52.385001 6166 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a8a434c22e45ca8faa53cb495af025. Latest consensus state: current_term: 1 leader_uuid: "d5a8a434c22e45ca8faa53cb495af025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } } }
I20260925 10:12:52.385107 6166 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:52.385334 6165 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a8a434c22e45ca8faa53cb495af025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a8a434c22e45ca8faa53cb495af025" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38239 } } }
I20260925 10:12:52.385680 6165 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:52.385833 6171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:52.386348 6171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:52.387903 6171 catalog_manager.cc:1347] Generated new cluster ID: 985c99d8e2674aeea6dba7c54c7f7cb1
I20260925 10:12:52.387977 6171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:52.392819 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.380466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.3.152.254:38239)
0925 10:12:52.380661 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:52.380670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:52.380706 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:12:52.382961 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:52.382968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:52.383047 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:52.383191 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:52.383844 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.385196 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.386351 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.386813 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.386934 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.389428 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:52.389433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:52.389433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:52.389452 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:52.391139 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.391517 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.391520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.391521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.391625 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.391850 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.391851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.391851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.392181 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:52.392371 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:52.392443 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:12:52.392585 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260925 10:12:52.400962 6171 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:52.401746 6171 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:52.404739 6171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a8a434c22e45ca8faa53cb495af025: Generated new TSK 0
I20260925 10:12:52.404917 6171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:52.435745 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:52.520015 6185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:52.520308 6185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:52.520354 6185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:52.521621 6185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:52.521697 6185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:52.521741 6185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:52.523226 6185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:52.523511 6185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:52.523757 6185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:52.528884 6190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.529036 6185 server_base.cc:1034] running on GCE node
W20260925 10:12:52.528882 6191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.529119 6193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.529361 6185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:52.529627 6185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:52.530727 6185 hybrid_clock.cc:648] HybridClock initialized: now 1790331172530712 us; error 30 us; skew 500 ppm
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:52.533403 6185 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:52.533864 6185 webserver.cc:460] Webserver started at http://127.3.152.193:40955/ using document root <none> and password file <none>
I20260925 10:12:52.534101 6185 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:52.534171 6185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:52.534327 6185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:52.535415 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "1975cb28cfc74c16b2235b86ee4d30ba"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "98fea90f09f244ac52417072358747c9"
server_key_iv: "6ee21966bc137dbe5a1eff98b476197e"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.535784 6185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "1975cb28cfc74c16b2235b86ee4d30ba"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "98fea90f09f244ac52417072358747c9"
server_key_iv: "6ee21966bc137dbe5a1eff98b476197e"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.537217 6185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:52.538020 6200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.538206 6185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:52.538316 6185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "1975cb28cfc74c16b2235b86ee4d30ba"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "98fea90f09f244ac52417072358747c9"
server_key_iv: "6ee21966bc137dbe5a1eff98b476197e"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.538403 6185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:52.587618 6185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:52.590152 6185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:52.590341 6185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:52.590613 6185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:52.590955 6185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:52.591012 6185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.591058 6185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:52.591105 6185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.597193 6185 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:33555
I20260925 10:12:52.597232 6314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:33555 every 8 connection(s)
I20260925 10:12:52.597615 6185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:52.602075 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6185
I20260925 10:12:52.602191 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:52.602366 3683 external_mini_cluster.cc:1428] Setting key b2d4832523d86e86786b5a581fad6de3
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:52.608619 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.598354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42651 (local address 127.3.152.254:38239)
0925 10:12:52.598444 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:52.598447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:52.598697 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:12:52.599194 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:52.599196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:52.599209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:52.599270 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:52.600265 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.600780 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.601921 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.602115 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.602201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.604882 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:52.604884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:52.604884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:52.604893 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:52.606808 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.607385 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.607387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.607387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.607451 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.607829 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.607830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.607830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.608015 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:52.608089 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:52.608471 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 10:12:52.608551 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:12:52.608911 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.598307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38239 (local address 127.3.152.193:42651)
0925 10:12:52.598497 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:52.598546 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:52.599103 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:52.599330 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:52.599334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:52.599938 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:52.600173 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.600179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.600878 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.600879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.601809 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.601812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.602210 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.602212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.602359 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.603002 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:52.603014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:52.604751 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:52.606910 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.606920 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.606923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.607287 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.607528 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.607529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.607530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.607700 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.608099 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:52.608107 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:52.608309 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 10:12:52.608443 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:12:52.608643 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260925 10:12:52.609550 6315 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38239
I20260925 10:12:52.609669 6315 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:52.609894 6315 heartbeater.cc:511] Master 127.3.152.254:38239 requested a full tablet report, sending...
I20260925 10:12:52.610479 6123 ts_manager.cc:194] Registered new tserver with Master: 1975cb28cfc74c16b2235b86ee4d30ba (127.3.152.193:33555)
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:12:52.611163 6123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:42651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:52.630717 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:52.712750 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:52.712970 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:52.712993 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:52.714202 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:52.714248 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:52.714275 6320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:52.715801 6320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:52.716081 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:52.716338 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:52.721616 6325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.721763 6326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.721750 6328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.721868 6320 server_base.cc:1034] running on GCE node
I20260925 10:12:52.722163 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:52.722440 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:52.723644 6320 hybrid_clock.cc:648] HybridClock initialized: now 1790331172723619 us; error 48 us; skew 500 ppm
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:52.726642 6320 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:52.727129 6320 webserver.cc:460] Webserver started at http://127.3.152.194:34781/ using document root <none> and password file <none>
I20260925 10:12:52.727376 6320 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:52.727447 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:52.727612 6320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:52.728792 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "0f8b1cca180e409f9ddd7e9f280f9d44"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "3c51fd59d80a07fae4fb0fce46c87f62"
server_key_iv: "3347825684c387a0c2212896fe9cebbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.729166 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "0f8b1cca180e409f9ddd7e9f280f9d44"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "3c51fd59d80a07fae4fb0fce46c87f62"
server_key_iv: "3347825684c387a0c2212896fe9cebbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.730546 6320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:12:52.731336 6335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.731540 6320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:52.731667 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "0f8b1cca180e409f9ddd7e9f280f9d44"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "3c51fd59d80a07fae4fb0fce46c87f62"
server_key_iv: "3347825684c387a0c2212896fe9cebbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.731745 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:52.747336 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:52.749814 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:52.749995 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:52.750264 6320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:52.750604 6320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:52.750662 6320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.750703 6320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:52.750758 6320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.756915 6320 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35699
I20260925 10:12:52.756954 6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35699 every 8 connection(s)
I20260925 10:12:52.757336 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:52.766024 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6320
I20260925 10:12:52.766157 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:52.766399 3683 external_mini_cluster.cc:1428] Setting key 167bd773f2202dd0ced125e46ce25548
I20260925 10:12:52.767491 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.758116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53513 (local address 127.3.152.254:38239)
0925 10:12:52.758164 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:52.758166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:52.758352 (+ 186us) server_negotiation.cc:407] Connection header received
0925 10:12:52.759379 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:52.759381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:52.759397 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:52.759452 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:52.760114 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.760689 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.761672 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.761885 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.761959 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.764259 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:52.764261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:52.764261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:52.764269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:52.765586 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.766107 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.766108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.766109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.766209 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.766651 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.766652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.766653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.766842 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:52.766918 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:52.767279 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:12:52.767395 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:12:52.767664 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.758096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38239 (local address 127.3.152.194:53513)
0925 10:12:52.758242 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:52.758279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:52.759268 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:52.759510 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:52.759514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:52.759770 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:52.760014 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.760021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.760772 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.760772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.761571 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.761573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.761989 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.761990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.762104 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.762689 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:52.762698 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:52.764141 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:52.765663 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.765670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.765672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.766005 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.766306 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.766307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.766307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.766469 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.766915 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:52.766921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:52.767089 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 10:12:52.767203 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:12:52.767387 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260925 10:12:52.768253 6449 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38239
I20260925 10:12:52.768401 6449 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:52.768627 6449 heartbeater.cc:511] Master 127.3.152.254:38239 requested a full tablet report, sending...
I20260925 10:12:52.769182 6123 ts_manager.cc:194] Registered new tserver with Master: 0f8b1cca180e409f9ddd7e9f280f9d44 (127.3.152.194:35699)
I20260925 10:12:52.769608 6123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:53513
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:52.795591 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:36459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:52.876952 6454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:52.877170 6454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:52.877194 6454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:52.878618 6454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:52.878664 6454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:52.878697 6454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:52.880203 6454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:36459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:52.880473 6454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:52.880759 6454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:52.886135 6460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:52.886106 6462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.886322 6454 server_base.cc:1034] running on GCE node
W20260925 10:12:52.886117 6459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:52.886576 6454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:52.886801 6454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:52.888006 6454 hybrid_clock.cc:648] HybridClock initialized: now 1790331172887984 us; error 24 us; skew 500 ppm
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:52.890487 6454 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:52.890924 6454 webserver.cc:460] Webserver started at http://127.3.152.195:39079/ using document root <none> and password file <none>
I20260925 10:12:52.891168 6454 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:52.891237 6454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:52.891389 6454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:52.892524 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "4c5e20c9c0014c97a4d4e32f11ab5a0e"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "35f6d015577c0a7eb28488dc9593263c"
server_key_iv: "d0d40637239e1a588527a03610300e88"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.892903 6454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "4c5e20c9c0014c97a4d4e32f11ab5a0e"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "35f6d015577c0a7eb28488dc9593263c"
server_key_iv: "d0d40637239e1a588527a03610300e88"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.894233 6454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:52.895012 6469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.895208 6454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:52.895316 6454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "4c5e20c9c0014c97a4d4e32f11ab5a0e"
format_stamp: "Formatted at 2026-09-25 10:12:52 on dist-test-slave-dzwq"
server_key: "35f6d015577c0a7eb28488dc9593263c"
server_key_iv: "d0d40637239e1a588527a03610300e88"
server_key_version: "encryptionkey@0"
I20260925 10:12:52.895392 6454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:52.911588 6454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:52.913645 6454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:52.913823 6454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:52.914068 6454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:52.914384 6454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:52.914440 6454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.914485 6454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:52.914507 6454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:52.920780 6454 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:34069
I20260925 10:12:52.920825 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:34069 every 8 connection(s)
I20260925 10:12:52.921146 6454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:12:52 dist-test-slave-dzwq krb5kdc[6078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331172, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:52.930142 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.921867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:34935 (local address 127.3.152.254:38239)
0925 10:12:52.921955 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:52.921958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:52.922167 (+ 209us) server_negotiation.cc:407] Connection header received
0925 10:12:52.922655 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:52.922656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:52.922666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:52.922728 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:52.923674 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.924247 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.925227 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.925416 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.925467 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.927742 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:52.927743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:52.927744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:52.927751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:52.928961 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.929340 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.929341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.929341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.929390 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:52.929631 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:52.929632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:52.929632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:52.929734 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:52.929785 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:52.930010 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:12:52.930070 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 10:12:52.930356 6586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:52.921808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38239 (local address 127.3.152.195:34935)
0925 10:12:52.922023 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:52.922066 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:52.922571 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:52.922800 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:52.922804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:52.923391 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:52.923594 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.923600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.924323 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.924324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.925150 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:52.925153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:52.925519 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:52.925520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:52.925675 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:52.926171 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:52.926179 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:52.927653 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:52.929031 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.929037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.929039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.929276 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.929447 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:52.929448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:52.929448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:52.929542 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:52.929787 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:52.929791 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:52.929900 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 10:12:52.929979 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 10:12:52.930094 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260925 10:12:52.930546 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6454
I20260925 10:12:52.930629 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:52.930878 3683 external_mini_cluster.cc:1428] Setting key 1fdcfa3f7d56205498aea2f6bfb90c16
I20260925 10:12:52.930979 6584 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38239
I20260925 10:12:52.931106 6584 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:52.931308 6584 heartbeater.cc:511] Master 127.3.152.254:38239 requested a full tablet report, sending...
I20260925 10:12:52.931658 6123 ts_manager.cc:194] Registered new tserver with Master: 4c5e20c9c0014c97a4d4e32f11ab5a0e (127.3.152.195:34069)
I20260925 10:12:52.932152 6123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:34935
I20260925 10:12:52.942762 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:12:52.954032 6163 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:12:52.953082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.3.152.254:38239)
0925 10:12:52.953183 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:52.953185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:52.953223 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:12:52.953328 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:52.953328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:52.953339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:52.953392 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:52.953962 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36222: BlockingRecv error: recv got EOF from 127.0.0.1:36222 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:12:52.954912 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6185
I20260925 10:12:52.959597 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6320
I20260925 10:12:52.963866 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6454
I20260925 10:12:52.968065 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6091
2026-09-25T10:12:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1883 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:53 dist-test-slave-dzwq krb5kdc[6596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:53 dist-test-slave-dzwq krb5kdc[6596](info): set up 2 sockets
Sep 25 10:12:53 dist-test-slave-dzwq krb5kdc[6596](info): commencing operation
krb5kdc: starting...
W20260925 10:12:54.008484 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:54.131680 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34295
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:54.215610 6609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:54.215920 6609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:54.215960 6609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:54.217217 6609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:54.217326 6609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:54.217355 6609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:54.217383 6609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:54.217415 6609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:54.219058 6609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:34295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:54.219352 6609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:54.219614 6609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:54.224992 6617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:54.225203 6609 server_base.cc:1034] running on GCE node
W20260925 10:12:54.224977 6615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:54.224974 6614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:54.225647 6609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:54.225961 6609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:54.227141 6609 hybrid_clock.cc:648] HybridClock initialized: now 1790331174227120 us; error 37 us; skew 500 ppm
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331174, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:54.234452 6609 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:54.235006 6609 webserver.cc:460] Webserver started at http://127.3.152.254:40439/ using document root <none> and password file <none>
I20260925 10:12:54.235296 6609 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:54.235348 6609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:54.235489 6609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:54.236703 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "461665cf67ba459f8ae86d305d1b326c"
format_stamp: "Formatted at 2026-09-25 10:12:54 on dist-test-slave-dzwq"
server_key: "ad53bd0cc82876a73bdb7c8e4162d802"
server_key_iv: "29e453cb5bf011fe75bcd7fc012a3834"
server_key_version: "encryptionkey@0"
I20260925 10:12:54.237082 6609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "461665cf67ba459f8ae86d305d1b326c"
format_stamp: "Formatted at 2026-09-25 10:12:54 on dist-test-slave-dzwq"
server_key: "ad53bd0cc82876a73bdb7c8e4162d802"
server_key_iv: "29e453cb5bf011fe75bcd7fc012a3834"
server_key_version: "encryptionkey@0"
I20260925 10:12:54.238557 6609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:54.239426 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:54.239699 6609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:54.239823 6609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "461665cf67ba459f8ae86d305d1b326c"
format_stamp: "Formatted at 2026-09-25 10:12:54 on dist-test-slave-dzwq"
server_key: "ad53bd0cc82876a73bdb7c8e4162d802"
server_key_iv: "29e453cb5bf011fe75bcd7fc012a3834"
server_key_version: "encryptionkey@0"
I20260925 10:12:54.239912 6609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:54.249083 6609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:54.251547 6609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:54.251724 6609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:54.256109 6609 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:34295
I20260925 10:12:54.256193 6676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:34295 every 8 connection(s)
I20260925 10:12:54.256579 6609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:54.257537 6677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:54.260183 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c: Bootstrap starting.
I20260925 10:12:54.260828 6677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:54.261145 6677 log.cc:824] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:54.261937 6677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c: No bootstrap required, opened a new log
I20260925 10:12:54.263734 6677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } }
I20260925 10:12:54.263882 6677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:54.263904 6677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461665cf67ba459f8ae86d305d1b326c, State: Initialized, Role: FOLLOWER
I20260925 10:12:54.264084 6677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } }
I20260925 10:12:54.264174 6677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:54.264201 6677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:54.264266 6677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:54.265132 6677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } }
I20260925 10:12:54.265223 6677 leader_election.cc:302] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461665cf67ba459f8ae86d305d1b326c; no voters:
I20260925 10:12:54.265419 6677 leader_election.cc:288] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:54.265532 6680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:54.265726 6680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [term 1 LEADER]: Becoming Leader. State: Replica: 461665cf67ba459f8ae86d305d1b326c, State: Running, Role: LEADER
I20260925 10:12:54.265743 6677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:54.265887 6680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } }
I20260925 10:12:54.266299 6681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461665cf67ba459f8ae86d305d1b326c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } } }
I20260925 10:12:54.266409 6681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:54.266297 6682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461665cf67ba459f8ae86d305d1b326c. Latest consensus state: current_term: 1 leader_uuid: "461665cf67ba459f8ae86d305d1b326c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461665cf67ba459f8ae86d305d1b326c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34295 } } }
I20260925 10:12:54.266805 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6609
I20260925 10:12:54.266902 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:54.267040 6688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:54.267128 3683 external_mini_cluster.cc:1428] Setting key 87799726e2025c8d11f156a46b48f228
I20260925 10:12:54.267570 6688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:54.268425 6682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:54.269515 6688 catalog_manager.cc:1347] Generated new cluster ID: 759d9cbe07bb4f2abe0f7e122b3dba72
I20260925 10:12:54.269567 6688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:54.279688 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:54.267786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.3.152.254:34295)
0925 10:12:54.267946 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:54.267985 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:54.268084 (+ 99us) server_negotiation.cc:407] Connection header received
0925 10:12:54.268387 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:54.268396 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:54.268465 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:54.268648 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:54.269473 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:54.270965 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:54.272055 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:54.272762 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:54.272881 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:54.275802 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:54.275807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:54.275808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:54.275822 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:54.277920 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:54.278368 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:54.278372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:54.278372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:54.278481 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:54.278705 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:54.278706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:54.278706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:54.279018 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:54.279174 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:54.279300 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 10:12:54.279438 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260925 10:12:54.280910 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:12:54.293996 6688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:54.294576 6688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:54.302100 6688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461665cf67ba459f8ae86d305d1b326c: Generated new TSK 0
I20260925 10:12:54.302335 6688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:12:54.302433 6699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:54.297664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.3.152.254:34295)
0925 10:12:54.297723 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:54.297725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:54.297941 (+ 216us) server_negotiation.cc:407] Connection header received
0925 10:12:54.298002 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:54.298003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:54.298016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:54.298071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:54.298494 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:54.299325 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:54.300425 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:54.300783 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:54.300861 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:54.301700 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:54.301702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:54.301702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:12:54.301969 (+ 267us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:54.301990 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:12:54.301999 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:54.302061 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:54.302314 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 10:12:54.302345 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:12:54.305119 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6609
2026-09-25T10:12:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1334 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6711](info): set up 2 sockets
Sep 25 10:12:54 dist-test-slave-dzwq krb5kdc[6711](info): commencing operation
krb5kdc: starting...
W20260925 10:12:55.388805 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:55.515861 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42759
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:42759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:55.602878 6724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:55.603096 6724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:55.603121 6724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:55.604277 6724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:55.604321 6724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:55.604336 6724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:55.604348 6724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:55.604360 6724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:55.605896 6724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:42759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:55.606110 6724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:55.606304 6724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:55.611363 6732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.611371 6730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.611363 6729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:55.611572 6724 server_base.cc:1034] running on GCE node
I20260925 10:12:55.612008 6724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:55.612332 6724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:55.613543 6724 hybrid_clock.cc:648] HybridClock initialized: now 1790331175613531 us; error 40 us; skew 500 ppm
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:55.616263 6724 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:55.616761 6724 webserver.cc:460] Webserver started at http://127.3.152.254:34965/ using document root <none> and password file <none>
I20260925 10:12:55.616972 6724 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:55.617015 6724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:55.617106 6724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:55.618175 6724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "b92392e945e24d84aceb25e98eccb045"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "f3b1865af68cdff5c85f1c9cc4e0007d"
server_key_iv: "527cbfcc4e557aad8f637c97377f0efa"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.618480 6724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "b92392e945e24d84aceb25e98eccb045"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "f3b1865af68cdff5c85f1c9cc4e0007d"
server_key_iv: "527cbfcc4e557aad8f637c97377f0efa"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.619881 6724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:55.620748 6739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:55.620972 6724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:55.621066 6724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "b92392e945e24d84aceb25e98eccb045"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "f3b1865af68cdff5c85f1c9cc4e0007d"
server_key_iv: "527cbfcc4e557aad8f637c97377f0efa"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.621146 6724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:55.628854 6724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:55.630879 6724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:55.631033 6724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:55.635097 6724 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:42759
I20260925 10:12:55.635126 6791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:42759 every 8 connection(s)
I20260925 10:12:55.635500 6724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:55.636554 6792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:55.638782 6792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045: Bootstrap starting.
I20260925 10:12:55.639320 6792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:55.639546 6792 log.cc:824] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:55.640254 6792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045: No bootstrap required, opened a new log
I20260925 10:12:55.640920 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6724
I20260925 10:12:55.641029 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:55.641198 3683 external_mini_cluster.cc:1428] Setting key d99bac70dca6f5dfe27536b6eeca2a57
I20260925 10:12:55.642392 6792 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } }
I20260925 10:12:55.642553 6792 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:55.642597 6792 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92392e945e24d84aceb25e98eccb045, State: Initialized, Role: FOLLOWER
I20260925 10:12:55.642747 6792 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } }
I20260925 10:12:55.642846 6792 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:55.642881 6792 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:55.642925 6792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:55.644030 6792 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } }
I20260925 10:12:55.644125 6792 leader_election.cc:302] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92392e945e24d84aceb25e98eccb045; no voters:
I20260925 10:12:55.644322 6792 leader_election.cc:288] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:55.644343 6797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:55.644455 6797 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [term 1 LEADER]: Becoming Leader. State: Replica: b92392e945e24d84aceb25e98eccb045, State: Running, Role: LEADER
I20260925 10:12:55.644539 6797 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } }
I20260925 10:12:55.644590 6792 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:55.644883 6798 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92392e945e24d84aceb25e98eccb045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } } }
I20260925 10:12:55.645133 6798 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:55.645049 6799 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92392e945e24d84aceb25e98eccb045. Latest consensus state: current_term: 1 leader_uuid: "b92392e945e24d84aceb25e98eccb045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92392e945e24d84aceb25e98eccb045" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42759 } } }
I20260925 10:12:55.645388 6799 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045 [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:55.645536 6807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:55.646075 6807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:55.647680 6807 catalog_manager.cc:1347] Generated new cluster ID: 0c2aeb186853454ba9e1a6ff7fa2b1b5
I20260925 10:12:55.647742 6807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:55.652839 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:55.641786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.3.152.254:42759)
0925 10:12:55.641996 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:55.642010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:55.642043 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:12:55.642401 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:55.642410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:55.642466 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:55.642717 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:55.643518 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.644858 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.646004 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.646603 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.646722 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:55.649322 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:55.649326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:55.649326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:55.649338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:55.651131 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:55.651537 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:55.651540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:55.651540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:55.651661 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:55.651878 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:55.651879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:55.651880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:55.652194 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:55.652376 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:55.652450 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:12:55.652618 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260925 10:12:55.655237 6807 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:55.655764 6807 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:55.662479 6807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92392e945e24d84aceb25e98eccb045: Generated new TSK 0
I20260925 10:12:55.662776 6807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:55.699628 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:55.797315 6819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:55.797571 6819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:55.797597 6819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:55.798789 6819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:55.798832 6819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:55.798872 6819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:55.800414 6819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:55.800697 6819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:55.800958 6819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:55.806577 6827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.806583 6825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.806593 6824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:55.806968 6819 server_base.cc:1034] running on GCE node
I20260925 10:12:55.807188 6819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:55.807451 6819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:55.808657 6819 hybrid_clock.cc:648] HybridClock initialized: now 1790331175808629 us; error 40 us; skew 500 ppm
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:55.811264 6819 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:55.811774 6819 webserver.cc:460] Webserver started at http://127.3.152.193:45055/ using document root <none> and password file <none>
I20260925 10:12:55.812103 6819 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:55.812178 6819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:55.812346 6819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:55.813506 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "9ad4ed225c294d53a0f84784dbc077c2"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "363fbc0282e3ab97538039928d4f9ec3"
server_key_iv: "92a0984573b4bbbdfc27f0241c89993a"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.813903 6819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "9ad4ed225c294d53a0f84784dbc077c2"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "363fbc0282e3ab97538039928d4f9ec3"
server_key_iv: "92a0984573b4bbbdfc27f0241c89993a"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.815361 6819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:55.816214 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:55.816421 6819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:55.816537 6819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "9ad4ed225c294d53a0f84784dbc077c2"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "363fbc0282e3ab97538039928d4f9ec3"
server_key_iv: "92a0984573b4bbbdfc27f0241c89993a"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.816617 6819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:55.843490 6819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:55.845856 6819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:55.846036 6819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:55.846355 6819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:55.846706 6819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:55.846745 6819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:55.846772 6819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:55.846818 6819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:55.852910 6819 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:38913
I20260925 10:12:55.852988 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:38913 every 8 connection(s)
I20260925 10:12:55.853359 6819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:55.856525 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6819
I20260925 10:12:55.856637 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:55.856834 3683 external_mini_cluster.cc:1428] Setting key 1c159628a8c981bd79aa13b8a765b4e9
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:55.863922 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:55.854192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42747 (local address 127.3.152.254:42759)
0925 10:12:55.854269 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:55.854271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:55.854467 (+ 196us) server_negotiation.cc:407] Connection header received
0925 10:12:55.855029 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:55.855031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:55.855048 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:55.855113 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:55.856138 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.856928 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.858154 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.858503 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.858580 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:55.860941 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:55.860942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:55.860943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:55.860951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:55.862453 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:55.862903 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:55.862905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:55.862906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:55.862966 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:55.863241 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:55.863242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:55.863242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:55.863364 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:55.863438 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:55.863729 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:12:55.863820 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:12:55.864166 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:55.854190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42759 (local address 127.3.152.193:42747)
0925 10:12:55.854334 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:55.854378 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:55.854914 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:55.855199 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:55.855203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:55.855792 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:55.856019 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:55.856024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.857117 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.857118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:55.858058 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:55.858060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:55.858590 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:55.858591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:55.858698 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:55.859316 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:55.859324 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:55.860853 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:55.862557 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:55.862565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:55.862568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:55.862811 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:55.863034 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:55.863035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:55.863035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:55.863163 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:55.863441 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:55.863449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:55.863597 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:12:55.863707 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:12:55.863889 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:12:55.864720 6949 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42759
I20260925 10:12:55.864858 6949 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:55.865046 6949 heartbeater.cc:511] Master 127.3.152.254:42759 requested a full tablet report, sending...
I20260925 10:12:55.865607 6756 ts_manager.cc:194] Registered new tserver with Master: 9ad4ed225c294d53a0f84784dbc077c2 (127.3.152.193:38913)
I20260925 10:12:55.866309 6756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:42747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:55.883244 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:55.966019 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:55.966248 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:55.966274 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:55.967473 6954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:55.967523 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:55.967551 6954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:55.969177 6954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:55.969455 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:55.969717 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:55.975114 6960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.975093 6962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:55.975076 6959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:55.975129 6954 server_base.cc:1034] running on GCE node
I20260925 10:12:55.975618 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:55.975890 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:55.977061 6954 hybrid_clock.cc:648] HybridClock initialized: now 1790331175977033 us; error 46 us; skew 500 ppm
Sep 25 10:12:55 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:55.979709 6954 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:55.980211 6954 webserver.cc:460] Webserver started at http://127.3.152.194:37265/ using document root <none> and password file <none>
I20260925 10:12:55.980432 6954 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:55.980475 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:55.980569 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:55.981635 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "645306daef154ca9b5508e7de2acc3ff"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "412617111fb9a65e0fe3ea6b183a0362"
server_key_iv: "38640b5f0e984b94cc1aa71e8e275753"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.981954 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "645306daef154ca9b5508e7de2acc3ff"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "412617111fb9a65e0fe3ea6b183a0362"
server_key_iv: "38640b5f0e984b94cc1aa71e8e275753"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.983387 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:55.984308 6969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:55.984547 6954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:55.984658 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "645306daef154ca9b5508e7de2acc3ff"
format_stamp: "Formatted at 2026-09-25 10:12:55 on dist-test-slave-dzwq"
server_key: "412617111fb9a65e0fe3ea6b183a0362"
server_key_iv: "38640b5f0e984b94cc1aa71e8e275753"
server_key_version: "encryptionkey@0"
I20260925 10:12:55.984742 6954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:55.997071 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:55.999244 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:55.999364 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:55.999593 6954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:55.999943 6954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:56.000041 6954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:56.000074 6954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:56.000093 6954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:56.006152 6954 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:39431
I20260925 10:12:56.006230 7083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:39431 every 8 connection(s)
I20260925 10:12:56.006574 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:12:56.008453 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 6954
I20260925 10:12:56.008576 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:56.008803 3683 external_mini_cluster.cc:1428] Setting key 6b0c3d3b35938c7425c9c04132102948
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[6711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331175, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:56.017714 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:56.007328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:49863 (local address 127.3.152.254:42759)
0925 10:12:56.007377 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:56.007380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:56.007569 (+ 189us) server_negotiation.cc:407] Connection header received
0925 10:12:56.008231 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:56.008232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:56.008246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:56.008304 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:56.009502 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.010000 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.011196 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.011416 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.011491 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:56.014308 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:56.014311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:56.014311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:56.014322 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:56.016166 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.016622 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.016624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.016625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.016700 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.016991 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.016993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.016993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.017121 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:56.017201 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:56.017523 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 10:12:56.017617 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:12:56.017916 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:56.007287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42759 (local address 127.3.152.194:49863)
0925 10:12:56.007426 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:56.007479 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:56.008118 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:56.008381 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:56.008385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:56.009192 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:56.009403 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:56.009409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.010097 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.010098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:56.011085 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:56.011088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.011513 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.011514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:56.011680 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:56.012363 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:56.012373 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:56.014188 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:56.016279 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:56.016287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:56.016290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:56.016538 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:56.016789 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:56.016790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:56.016790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:56.016915 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:56.017190 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:56.017197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:56.017379 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 10:12:56.017492 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:12:56.017670 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:12:56.018560 7084 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42759
I20260925 10:12:56.018721 7084 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:56.018958 7084 heartbeater.cc:511] Master 127.3.152.254:42759 requested a full tablet report, sending...
I20260925 10:12:56.019594 6756 ts_manager.cc:194] Registered new tserver with Master: 645306daef154ca9b5508e7de2acc3ff (127.3.152.194:39431)
I20260925 10:12:56.020061 6756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:49863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:56.039871 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:56.128940 7089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:56.129204 7089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:56.129246 7089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:56.130458 7089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:56.130537 7089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:56.130586 7089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:56.132272 7089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:56.132570 7089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:56.132824 7089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:56.138084 7095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:56.138100 7097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:56.138103 7094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:56.138353 7089 server_base.cc:1034] running on GCE node
I20260925 10:12:56.138566 7089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:56.138880 7089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:56.140060 7089 hybrid_clock.cc:648] HybridClock initialized: now 1790331176140012 us; error 48 us; skew 500 ppm
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331176, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:56.142874 7089 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:56.143362 7089 webserver.cc:460] Webserver started at http://127.3.152.195:41275/ using document root <none> and password file <none>
I20260925 10:12:56.143612 7089 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:56.143682 7089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:56.143832 7089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:56.145179 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "f7d8181eb62f4f27a92b25d246c16e1b"
format_stamp: "Formatted at 2026-09-25 10:12:56 on dist-test-slave-dzwq"
server_key: "62847ef74606cb68ce1775e463dbdee2"
server_key_iv: "25490dd06e167ae3a917ad0a252edbbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:56.145566 7089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "f7d8181eb62f4f27a92b25d246c16e1b"
format_stamp: "Formatted at 2026-09-25 10:12:56 on dist-test-slave-dzwq"
server_key: "62847ef74606cb68ce1775e463dbdee2"
server_key_iv: "25490dd06e167ae3a917ad0a252edbbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:56.147030 7089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:56.147801 7104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:56.148031 7089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:56.148160 7089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "f7d8181eb62f4f27a92b25d246c16e1b"
format_stamp: "Formatted at 2026-09-25 10:12:56 on dist-test-slave-dzwq"
server_key: "62847ef74606cb68ce1775e463dbdee2"
server_key_iv: "25490dd06e167ae3a917ad0a252edbbc"
server_key_version: "encryptionkey@0"
I20260925 10:12:56.148238 7089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:56.159613 7089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:56.162168 7089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:56.162402 7089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:56.162698 7089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:56.163105 7089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:56.163153 7089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:56.163213 7089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:56.163241 7089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:56.169461 7089 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:37645
I20260925 10:12:56.169526 7217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:37645 every 8 connection(s)
I20260925 10:12:56.169858 7089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:56.175684 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7089
I20260925 10:12:56.175815 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:56.176065 3683 external_mini_cluster.cc:1428] Setting key 48ae54dd6c2ce142e43d5fce49f1f4c8
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[6711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331176, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:56.180454 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:56.170641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:57677 (local address 127.3.152.254:42759)
0925 10:12:56.170709 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:56.170712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:56.170905 (+ 193us) server_negotiation.cc:407] Connection header received
0925 10:12:56.171478 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:56.171480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:56.171493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:56.171551 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:56.172643 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.173150 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.174196 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.174399 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.174488 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:56.177252 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:56.177254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:56.177254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:56.177263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:56.178826 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.179301 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.179302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.179303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.179381 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.179652 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.179653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.179654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.179793 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:56.179872 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:56.180253 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 10:12:56.180342 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:12:56.180665 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:56.170595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42759 (local address 127.3.152.195:57677)
0925 10:12:56.170764 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:56.170816 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:56.171366 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:56.171627 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:56.171631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:56.172321 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:56.172529 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:56.172534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.173259 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.173260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:56.174091 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:56.174093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.174501 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.174502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:56.174632 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:56.175153 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:56.175161 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:56.177154 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:56.178948 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:56.178956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:56.178959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:56.179215 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:56.179452 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:56.179453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:56.179453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:56.179580 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:56.179873 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:56.179881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:56.180111 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 10:12:56.180230 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 10:12:56.180405 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260925 10:12:56.181305 7218 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42759
I20260925 10:12:56.181422 7218 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:56.181623 7218 heartbeater.cc:511] Master 127.3.152.254:42759 requested a full tablet report, sending...
I20260925 10:12:56.182047 6756 ts_manager.cc:194] Registered new tserver with Master: f7d8181eb62f4f27a92b25d246c16e1b (127.3.152.195:37645)
I20260925 10:12:56.182404 6756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:57677
I20260925 10:12:56.188179 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[6711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331176, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[6711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331176, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:56.211720 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:56.204245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.3.152.254:42759)
0925 10:12:56.204318 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:56.204321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:56.204532 (+ 211us) server_negotiation.cc:407] Connection header received
0925 10:12:56.204628 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:56.204629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:56.204640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:56.204702 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:56.205345 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.205858 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.206650 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:56.206846 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:56.206897 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:56.209328 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:56.209329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:56.209330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:56.209338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:56.210754 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.211074 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.211075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.211075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.211126 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:56.211288 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:56.211289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:56.211289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:56.211433 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:56.211508 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:56.211594 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:12:56.211652 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 10:12:56.212059 6756 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58016
I20260925 10:12:56.212563 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6819
I20260925 10:12:56.217180 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6954
I20260925 10:12:56.221524 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7089
I20260925 10:12:56.225831 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 6724
2026-09-25T10:12:56Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1923 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[7231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[7231](info): set up 2 sockets
Sep 25 10:12:56 dist-test-slave-dzwq krb5kdc[7231](info): commencing operation
krb5kdc: starting...
W20260925 10:12:57.276871 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:57.403295 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35369
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:35369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:57.488495 7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:57.488768 7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:57.488811 7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:57.490010 7244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:57.490080 7244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:57.490105 7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:57.490126 7244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:57.490159 7244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:57.491779 7244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:35369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:57.492092 7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:57.492337 7244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:57.497520 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:57.497511 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:57.497511 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:57.497545 7244 server_base.cc:1034] running on GCE node
I20260925 10:12:57.498034 7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:57.498353 7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:57.499538 7244 hybrid_clock.cc:648] HybridClock initialized: now 1790331177499523 us; error 45 us; skew 500 ppm
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:57.502308 7244 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:57.502868 7244 webserver.cc:460] Webserver started at http://127.3.152.254:44153/ using document root <none> and password file <none>
I20260925 10:12:57.503127 7244 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:57.503199 7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:57.503352 7244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:57.504519 7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "91eb916e7eba4876b2387288c1d6ee8c"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "b8c44e9f84b96d795933778a442f9c8a"
server_key_iv: "f1120d1e0f0700cc4c4036c1eceb4099"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.504904 7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "91eb916e7eba4876b2387288c1d6ee8c"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "b8c44e9f84b96d795933778a442f9c8a"
server_key_iv: "f1120d1e0f0700cc4c4036c1eceb4099"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.506263 7244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:57.507015 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.507225 7244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:57.507335 7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "91eb916e7eba4876b2387288c1d6ee8c"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "b8c44e9f84b96d795933778a442f9c8a"
server_key_iv: "f1120d1e0f0700cc4c4036c1eceb4099"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.507414 7244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:57.520226 7244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:57.522204 7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:57.522704 7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:57.527192 7244 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:35369
I20260925 10:12:57.527223 7312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:35369 every 8 connection(s)
I20260925 10:12:57.527767 7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:57.528515 7313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:57.528728 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7244
I20260925 10:12:57.528847 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:57.529052 3683 external_mini_cluster.cc:1428] Setting key 92ee64b5ae93475373195da06e05b6a0
I20260925 10:12:57.532178 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c: Bootstrap starting.
I20260925 10:12:57.532923 7313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:57.533238 7313 log.cc:824] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:57.533918 7313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c: No bootstrap required, opened a new log
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:57.536195 7313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } }
I20260925 10:12:57.536371 7313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:57.536410 7313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91eb916e7eba4876b2387288c1d6ee8c, State: Initialized, Role: FOLLOWER
I20260925 10:12:57.536542 7313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } }
I20260925 10:12:57.536602 7313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:57.536624 7313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:57.536651 7313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:57.537230 7313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } }
I20260925 10:12:57.537360 7313 leader_election.cc:302] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91eb916e7eba4876b2387288c1d6ee8c; no voters:
I20260925 10:12:57.537576 7313 leader_election.cc:288] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:57.537693 7318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:57.537884 7318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [term 1 LEADER]: Becoming Leader. State: Replica: 91eb916e7eba4876b2387288c1d6ee8c, State: Running, Role: LEADER
I20260925 10:12:57.537899 7313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:57.538323 7318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } }
I20260925 10:12:57.538736 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91eb916e7eba4876b2387288c1d6ee8c. Latest consensus state: current_term: 1 leader_uuid: "91eb916e7eba4876b2387288c1d6ee8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } } }
I20260925 10:12:57.538827 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:57.539291 7327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:57.539274 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91eb916e7eba4876b2387288c1d6ee8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91eb916e7eba4876b2387288c1d6ee8c" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35369 } } }
I20260925 10:12:57.539425 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:57.539921 7327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:57.541076 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:57.529589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.3.152.254:35369)
0925 10:12:57.529779 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:57.529789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:57.529823 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:12:57.530334 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:57.530343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:57.530411 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:57.530587 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:57.531287 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.532722 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.533592 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.534085 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.534260 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:57.537073 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:57.537077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:57.537077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:57.537088 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:57.539348 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.539819 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.539822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.539823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.539938 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.540238 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.540239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.540239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.540484 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:57.540644 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:57.540762 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:12:57.540908 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260925 10:12:57.541715 7327 catalog_manager.cc:1347] Generated new cluster ID: dd9feed57b8a4d0e824d3882f3465b88
I20260925 10:12:57.541765 7327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:57.565310 7327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:57.566074 7327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:57.569032 7327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91eb916e7eba4876b2387288c1d6ee8c: Generated new TSK 0
I20260925 10:12:57.569254 7327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:57.607122 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:57.711396 7339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:57.711615 7339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:57.711643 7339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:57.712913 7339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:57.712960 7339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:57.712987 7339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:57.714550 7339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:57.714814 7339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:57.715072 7339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:57.720481 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:57.720481 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:57.720507 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:57.720670 7339 server_base.cc:1034] running on GCE node
I20260925 10:12:57.720893 7339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:57.721151 7339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:57.722321 7339 hybrid_clock.cc:648] HybridClock initialized: now 1790331177722312 us; error 42 us; skew 500 ppm
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:57.729111 7339 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:57.729600 7339 webserver.cc:460] Webserver started at http://127.3.152.193:39013/ using document root <none> and password file <none>
I20260925 10:12:57.729840 7339 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:57.729912 7339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:57.730070 7339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:57.731182 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "c889d7caad7d479f87b12605742d46e9"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "4fd634cf29402f731dcf093dd0527373"
server_key_iv: "02e67f816c82fcc3e5c8b6d601879017"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.731563 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "c889d7caad7d479f87b12605742d46e9"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "4fd634cf29402f731dcf093dd0527373"
server_key_iv: "02e67f816c82fcc3e5c8b6d601879017"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.733062 7339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:12:57.733852 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.734035 7339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:57.734138 7339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "c889d7caad7d479f87b12605742d46e9"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "4fd634cf29402f731dcf093dd0527373"
server_key_iv: "02e67f816c82fcc3e5c8b6d601879017"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.734225 7339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:57.758500 7339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:57.760540 7339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:57.760725 7339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:57.760999 7339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:57.761343 7339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:57.761400 7339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.761443 7339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:57.761473 7339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.767546 7339 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34059
I20260925 10:12:57.767598 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34059 every 8 connection(s)
I20260925 10:12:57.767935 7339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:57.773512 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7339
I20260925 10:12:57.773625 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:57.774070 3683 external_mini_cluster.cc:1428] Setting key 65fc1ee5036a055937e52317fa785959
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:57.780120 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:57.768726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:55839 (local address 127.3.152.254:35369)
0925 10:12:57.768789 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:57.768792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:57.769025 (+ 233us) server_negotiation.cc:407] Connection header received
0925 10:12:57.769932 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:57.769933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:57.769942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:57.770004 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:57.770631 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.771136 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.772621 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.772852 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.772933 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:57.776219 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:57.776221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:57.776221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:57.776230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:57.778307 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.778888 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.778889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.778889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.778951 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.779231 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.779231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.779232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.779388 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:57.779433 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:57.779844 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 10:12:57.779955 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:12:57.780773 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:57.768687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35369 (local address 127.3.152.193:55839)
0925 10:12:57.768865 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:57.768909 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:57.769837 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:57.770076 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:57.770080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:57.770348 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:57.770548 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:57.770552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.771221 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.771222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:57.772525 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:57.772528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.772955 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.772956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:57.773106 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:57.773902 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:57.773913 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:57.776107 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:57.778415 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:57.778424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:57.778427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:57.778797 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:57.779020 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:57.779020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:57.779021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:57.779164 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:57.779481 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:57.779488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:57.779667 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 10:12:57.780276 (+ 609us) client_negotiation.cc:240] Negotiation successful
0925 10:12:57.780491 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260925 10:12:57.781492 7469 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35369
I20260925 10:12:57.781610 7469 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:57.781847 7469 heartbeater.cc:511] Master 127.3.152.254:35369 requested a full tablet report, sending...
I20260925 10:12:57.782449 7276 ts_manager.cc:194] Registered new tserver with Master: c889d7caad7d479f87b12605742d46e9 (127.3.152.193:34059)
I20260925 10:12:57.783331 7276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:55839
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:57.807693 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:57.897313 7474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:57.897534 7474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:57.897559 7474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:57.898774 7474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:57.898820 7474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:57.898849 7474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:57.900471 7474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:57.900725 7474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:57.900969 7474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:57.906221 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:57.906229 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:57.906253 7474 server_base.cc:1034] running on GCE node
W20260925 10:12:57.906224 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:57.906752 7474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:57.906958 7474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:57.908111 7474 hybrid_clock.cc:648] HybridClock initialized: now 1790331177908091 us; error 37 us; skew 500 ppm
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:57.910815 7474 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:57.911267 7474 webserver.cc:460] Webserver started at http://127.3.152.194:42483/ using document root <none> and password file <none>
I20260925 10:12:57.911473 7474 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:57.911514 7474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:57.911605 7474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:57.912695 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "4c3b5e52245d4d508c92a63de42f087d"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "02945fc9affc817a89ff9e2c2bc2ff7a"
server_key_iv: "c5b53a30aa2110e0891a1b874ed8e08c"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.913028 7474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "4c3b5e52245d4d508c92a63de42f087d"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "02945fc9affc817a89ff9e2c2bc2ff7a"
server_key_iv: "c5b53a30aa2110e0891a1b874ed8e08c"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.914446 7474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:12:57.915140 7489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.915333 7474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:57.915424 7474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "4c3b5e52245d4d508c92a63de42f087d"
format_stamp: "Formatted at 2026-09-25 10:12:57 on dist-test-slave-dzwq"
server_key: "02945fc9affc817a89ff9e2c2bc2ff7a"
server_key_iv: "c5b53a30aa2110e0891a1b874ed8e08c"
server_key_version: "encryptionkey@0"
I20260925 10:12:57.915498 7474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:57.926262 7474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:57.928295 7474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:57.928460 7474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:57.928695 7474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:57.929020 7474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:57.929076 7474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.929117 7474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:57.929145 7474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:57.935042 7474 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:33385
I20260925 10:12:57.935084 7604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:33385 every 8 connection(s)
I20260925 10:12:57.935431 7474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:57 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331177, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:57.944278 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7474
I20260925 10:12:57.944406 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:57.944674 3683 external_mini_cluster.cc:1428] Setting key 28be75e385d6ab50a3d5b40601e8d550
I20260925 10:12:57.945549 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:57.936274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:50701 (local address 127.3.152.254:35369)
0925 10:12:57.936348 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:57.936350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:57.936676 (+ 326us) server_negotiation.cc:407] Connection header received
0925 10:12:57.937872 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:57.937874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:57.937884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:57.937942 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:57.938792 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.939358 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.940374 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.940557 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.940610 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:57.942613 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:57.942615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:57.942615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:57.942621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:57.943877 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.944416 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.944417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.944418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.944506 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:57.944825 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:57.944825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:57.944825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:57.944993 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:57.945068 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:57.945376 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 10:12:57.945469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:12:57.945784 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:57.936249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35369 (local address 127.3.152.194:50701)
0925 10:12:57.936480 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:57.936519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:57.937775 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:57.938065 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:57.938069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:57.938479 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:57.938695 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:57.938699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.939445 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.939446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:57.940292 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:57.940294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:57.940628 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:57.940628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:57.940723 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:57.941191 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:57.941200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:57.942531 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:57.943953 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:57.943959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:57.944003 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:57.944335 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:57.944592 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:57.944593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:57.944594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:57.944762 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:57.945059 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:57.945065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:57.945199 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 10:12:57.945327 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:12:57.945511 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260925 10:12:57.946228 7605 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35369
I20260925 10:12:57.946334 7605 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:57.946556 7605 heartbeater.cc:511] Master 127.3.152.254:35369 requested a full tablet report, sending...
I20260925 10:12:57.946985 7276 ts_manager.cc:194] Registered new tserver with Master: 4c3b5e52245d4d508c92a63de42f087d (127.3.152.194:33385)
I20260925 10:12:57.947373 7276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:50701
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:57.970983 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:58.054167 7610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:58.054416 7610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:58.054443 7610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:58.055680 7610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:58.055730 7610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:58.055768 7610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:58.057459 7610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:58.057744 7610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:58.058001 7610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:58.063201 7615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:58.063234 7618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:58.063206 7616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:58.063609 7610 server_base.cc:1034] running on GCE node
I20260925 10:12:58.063856 7610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:58.064167 7610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:58.065351 7610 hybrid_clock.cc:648] HybridClock initialized: now 1790331178065335 us; error 31 us; skew 500 ppm
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331178, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:58.068049 7610 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:58.068569 7610 webserver.cc:460] Webserver started at http://127.3.152.195:34569/ using document root <none> and password file <none>
I20260925 10:12:58.068830 7610 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:58.068877 7610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:58.069023 7610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:58.070194 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "204cc19def6743cea809ae9843f04999"
format_stamp: "Formatted at 2026-09-25 10:12:58 on dist-test-slave-dzwq"
server_key: "2e9971c927c2e4cd216b93d452b634c1"
server_key_iv: "d74b9620b8224fb9cf3b553a566ef47c"
server_key_version: "encryptionkey@0"
I20260925 10:12:58.070509 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "204cc19def6743cea809ae9843f04999"
format_stamp: "Formatted at 2026-09-25 10:12:58 on dist-test-slave-dzwq"
server_key: "2e9971c927c2e4cd216b93d452b634c1"
server_key_iv: "d74b9620b8224fb9cf3b553a566ef47c"
server_key_version: "encryptionkey@0"
I20260925 10:12:58.071800 7610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:12:58.072584 7626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:58.072803 7610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:12:58.072901 7610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "204cc19def6743cea809ae9843f04999"
format_stamp: "Formatted at 2026-09-25 10:12:58 on dist-test-slave-dzwq"
server_key: "2e9971c927c2e4cd216b93d452b634c1"
server_key_iv: "d74b9620b8224fb9cf3b553a566ef47c"
server_key_version: "encryptionkey@0"
I20260925 10:12:58.072978 7610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:58.121865 7610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:58.123919 7610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:58.124136 7610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:58.124408 7610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:58.124787 7610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:58.124823 7610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:58.124868 7610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:58.124895 7610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:58.130761 7610 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:39937
I20260925 10:12:58.130815 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:39937 every 8 connection(s)
I20260925 10:12:58.131134 7610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:12:58.136659 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7610
I20260925 10:12:58.136780 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:12:58.137007 3683 external_mini_cluster.cc:1428] Setting key 04b35be30de8cee70b41b9fe789c1eeb
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331178, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:58.141309 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:58.131842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:54177 (local address 127.3.152.254:35369)
0925 10:12:58.131916 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:58.131918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:58.132185 (+ 267us) server_negotiation.cc:407] Connection header received
0925 10:12:58.133148 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:58.133149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:58.133160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:58.133215 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:58.133873 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.134364 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.135380 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.135561 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.135615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:58.138561 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:58.138562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:58.138563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:58.138573 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:58.140106 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.140489 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.140490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.140490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.140541 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.140764 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.140765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.140765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.140871 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:58.140922 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:58.141166 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 10:12:58.141239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:12:58.141543 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:58.131823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35369 (local address 127.3.152.195:54177)
0925 10:12:58.132027 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:58.132072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:58.133045 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:58.133305 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:58.133308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:58.133581 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:58.133791 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:58.133795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.134449 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.134450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:58.135296 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:58.135299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.135639 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.135640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:58.135771 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:58.136384 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:58.136393 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:58.138447 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:58.140181 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:58.140188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:58.140190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:58.140424 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:58.140597 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:58.140597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:58.140597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:58.140704 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:58.140927 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:58.140932 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:58.141053 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 10:12:58.141138 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:12:58.141276 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260925 10:12:58.142099 7741 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35369
I20260925 10:12:58.142205 7741 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:58.142408 7741 heartbeater.cc:511] Master 127.3.152.254:35369 requested a full tablet report, sending...
I20260925 10:12:58.142794 7276 ts_manager.cc:194] Registered new tserver with Master: 204cc19def6743cea809ae9843f04999 (127.3.152.195:39937)
I20260925 10:12:58.143172 7276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:54177
I20260925 10:12:58.148674 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:12:58.171300 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:58.163688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.3.152.193:34059)
0925 10:12:58.163883 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:58.163887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:58.163898 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:12:58.163925 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:58.163927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:58.163940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:58.164055 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:58.164617 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.165364 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.166092 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.166406 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.166460 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:58.168749 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:58.168758 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:58.168759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:58.168766 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:58.170189 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.170511 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.170514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.170514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.170582 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.170747 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.170747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.170747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.170876 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:58.170980 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:58.171091 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 10:12:58.171196 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7231](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:58.179042 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:58.172739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.3.152.254:35369)
0925 10:12:58.172797 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:58.172800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:58.172810 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:12:58.172836 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:58.172837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:58.172853 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:58.172902 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:58.173350 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.173834 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.174478 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:58.174654 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:58.174703 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:58.176911 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:58.176913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:58.176913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:58.176919 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:58.178107 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.178409 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.178409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.178410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.178457 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:58.178611 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:58.178612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:58.178612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:58.178749 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:58.178834 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:58.178917 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:12:58.178976 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 10:12:58.179298 7276 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52678
I20260925 10:12:58.179701 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7339
I20260925 10:12:58.184515 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7474
I20260925 10:12:58.188974 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7610
I20260925 10:12:58.193292 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7244
2026-09-25T10:12:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1967 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7753](info): set up 2 sockets
Sep 25 10:12:58 dist-test-slave-dzwq krb5kdc[7753](info): commencing operation
krb5kdc: starting...
W20260925 10:12:59.244717 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:12:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:12:59.367539 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41025
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:59.454843 7766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:59.455096 7766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:59.455135 7766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:59.456317 7766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:12:59.456384 7766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:59.456409 7766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:59.456439 7766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:12:59.456470 7766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:12:59.458010 7766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:59.458302 7766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:59.458588 7766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:59.463477 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.463496 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.463490 7772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:59.463722 7766 server_base.cc:1034] running on GCE node
I20260925 10:12:59.463949 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:59.464313 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:59.465513 7766 hybrid_clock.cc:648] HybridClock initialized: now 1790331179465495 us; error 29 us; skew 500 ppm
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:59.472034 7766 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:12:59.472397 7766 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:59.472447 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:59.472543 7766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:59.473610 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "bcbd98b4ed30484181822759383f342d"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "e2f8838f9b9fbefb985feebd9f313b61"
server_key_iv: "48bc4399d1a54782c1070c2125bd2405"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.473922 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "bcbd98b4ed30484181822759383f342d"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "e2f8838f9b9fbefb985feebd9f313b61"
server_key_iv: "48bc4399d1a54782c1070c2125bd2405"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.475178 7766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:12:59.475931 7780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.476127 7766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:59.476230 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "bcbd98b4ed30484181822759383f342d"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "e2f8838f9b9fbefb985feebd9f313b61"
server_key_iv: "48bc4399d1a54782c1070c2125bd2405"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.476317 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:59.484192 7766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:59.486579 7766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:59.486755 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:59.490922 7766 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41025
I20260925 10:12:59.490940 7832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41025 every 8 connection(s)
I20260925 10:12:59.491262 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:12:59.492451 7833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:12:59.492810 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7766
I20260925 10:12:59.492928 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:12:59.493119 3683 external_mini_cluster.cc:1428] Setting key c8d2a9a5b1b594d1b275c497b51b114b
I20260925 10:12:59.495798 7833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: Bootstrap starting.
I20260925 10:12:59.496495 7833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: Neither blocks nor log segments found. Creating new log.
I20260925 10:12:59.496840 7833 log.cc:824] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:12:59.497663 7833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: No bootstrap required, opened a new log
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:59.500294 7833 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } }
I20260925 10:12:59.500432 7833 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:12:59.500499 7833 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbd98b4ed30484181822759383f342d, State: Initialized, Role: FOLLOWER
I20260925 10:12:59.500648 7833 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } }
I20260925 10:12:59.500710 7833 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:12:59.500731 7833 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:12:59.500801 7833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:12:59.501390 7833 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } }
I20260925 10:12:59.501479 7833 leader_election.cc:302] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbd98b4ed30484181822759383f342d; no voters:
I20260925 10:12:59.501693 7833 leader_election.cc:288] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:12:59.501832 7838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:12:59.501969 7838 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [term 1 LEADER]: Becoming Leader. State: Replica: bcbd98b4ed30484181822759383f342d, State: Running, Role: LEADER
I20260925 10:12:59.501977 7833 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:12:59.502102 7838 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } }
I20260925 10:12:59.502835 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbd98b4ed30484181822759383f342d. Latest consensus state: current_term: 1 leader_uuid: "bcbd98b4ed30484181822759383f342d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } } }
I20260925 10:12:59.502909 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:59.503027 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbd98b4ed30484181822759383f342d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbd98b4ed30484181822759383f342d" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41025 } } }
I20260925 10:12:59.503103 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d [sys.catalog]: This master's current role is: LEADER
I20260925 10:12:59.504150 7848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:12:59.504711 7848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:12:59.505412 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:59.493700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.3.152.254:41025)
0925 10:12:59.493897 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:59.493914 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:59.493940 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:12:59.494386 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:59.494394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:59.494442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:59.494566 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:59.495392 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.496485 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.497534 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.498015 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.498100 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:59.501086 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:59.501109 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:59.501109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:59.501120 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:59.503450 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.503918 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.503921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.503921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.504063 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.504340 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.504341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.504341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.504713 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:59.504883 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:59.505060 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 10:12:59.505205 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260925 10:12:59.506546 7848 catalog_manager.cc:1347] Generated new cluster ID: 3fa0b7806d724acdbd2ccec4e384e9e0
I20260925 10:12:59.506608 7848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:12:59.510495 7848 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:12:59.511062 7848 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:12:59.519292 7848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: Generated new TSK 0
I20260925 10:12:59.519516 7848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:12:59.555289 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:59.640769 7860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:59.641000 7860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:59.641037 7860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:59.642282 7860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:59.642359 7860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:59.642403 7860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:12:59.643920 7860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:59.644223 7860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:59.644471 7860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:59.649533 7866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.649626 7868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.649546 7865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:59.649804 7860 server_base.cc:1034] running on GCE node
I20260925 10:12:59.650041 7860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:59.650296 7860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:59.651459 7860 hybrid_clock.cc:648] HybridClock initialized: now 1790331179651443 us; error 26 us; skew 500 ppm
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:59.654035 7860 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:12:59.654386 7860 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:59.654460 7860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:59.654610 7860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:59.655663 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "b44c7ea869894b27a4dfeef18945453a"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a6b38c9ce2edc7eab4e12da25da8aa5d"
server_key_iv: "b787d6ccc239a9d97ae70043a316f48c"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.656102 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "b44c7ea869894b27a4dfeef18945453a"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a6b38c9ce2edc7eab4e12da25da8aa5d"
server_key_iv: "b787d6ccc239a9d97ae70043a316f48c"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.657419 7860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:12:59.658152 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.658330 7860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:12:59.658447 7860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "b44c7ea869894b27a4dfeef18945453a"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a6b38c9ce2edc7eab4e12da25da8aa5d"
server_key_iv: "b787d6ccc239a9d97ae70043a316f48c"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.658521 7860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:59.677446 7860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:59.679949 7860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:59.680200 7860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:59.680487 7860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:59.680828 7860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:59.680884 7860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.680929 7860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:59.680971 7860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.687077 7860 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:40303
I20260925 10:12:59.687173 7988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:40303 every 8 connection(s)
I20260925 10:12:59.687440 7860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:12:59.691517 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7860
I20260925 10:12:59.691669 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:12:59.691890 3683 external_mini_cluster.cc:1428] Setting key 8c99a6b6c8c7edc09ecb078877828077
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:59.698426 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:59.688378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:58937 (local address 127.3.152.254:41025)
0925 10:12:59.688426 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:59.688428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:59.688646 (+ 218us) server_negotiation.cc:407] Connection header received
0925 10:12:59.689186 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:59.689188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:59.689201 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:59.689269 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:59.690261 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.690771 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.691858 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.692198 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.692279 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:59.694814 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:59.694815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:59.694816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:59.694823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:59.696760 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.697229 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.697230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.697231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.697310 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.697563 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.697564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.697564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.697727 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:59.697820 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:59.698175 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 10:12:59.698284 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:12:59.698477 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:59.688329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41025 (local address 127.3.152.193:58937)
0925 10:12:59.688520 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:59.688558 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:59.689076 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:59.689371 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:59.689376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:59.689645 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:59.690138 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:59.690143 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.690870 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.690871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:59.691784 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:59.691788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.692290 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.692291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:59.692433 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:59.693108 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:59.693117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:59.694668 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:59.696863 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:59.696873 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:59.696876 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:59.697147 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:59.697387 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:59.697388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:59.697388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:59.697496 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:59.697871 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:59.697878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:59.698020 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 10:12:59.698130 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:12:59.698275 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260925 10:12:59.699157 7989 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41025
I20260925 10:12:59.699283 7989 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:59.699440 7989 heartbeater.cc:511] Master 127.3.152.254:41025 requested a full tablet report, sending...
I20260925 10:12:59.700065 7797 ts_manager.cc:194] Registered new tserver with Master: b44c7ea869894b27a4dfeef18945453a (127.3.152.193:40303)
I20260925 10:12:59.700734 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:58937
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:12:59.722847 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:59.805580 7994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:59.805841 7994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:59.805884 7994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:59.807106 7994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:59.807178 7994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:59.807221 7994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:12:59.808873 7994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:59.809182 7994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:59.809468 7994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:59.814631 7999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.814769 8000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.814798 8002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:59.814980 7994 server_base.cc:1034] running on GCE node
I20260925 10:12:59.815222 7994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:59.815480 7994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:59.816641 7994 hybrid_clock.cc:648] HybridClock initialized: now 1790331179816661 us; error 64 us; skew 500 ppm
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:59.819213 7994 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:12:59.819628 7994 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:59.819700 7994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:59.819849 7994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:59.820983 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "aa5663677b584e9ea587c039335e8dfc"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "1cd62a332b5a3ff89cd7c639fa2e3edf"
server_key_iv: "0ad4ab3a269ba4cc4e8284e6b4017e7e"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.821365 7994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "aa5663677b584e9ea587c039335e8dfc"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "1cd62a332b5a3ff89cd7c639fa2e3edf"
server_key_iv: "0ad4ab3a269ba4cc4e8284e6b4017e7e"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.822723 7994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:12:59.823522 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.823745 7994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:12:59.823845 7994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "aa5663677b584e9ea587c039335e8dfc"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "1cd62a332b5a3ff89cd7c639fa2e3edf"
server_key_iv: "0ad4ab3a269ba4cc4e8284e6b4017e7e"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.823930 7994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:12:59.840677 7994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:12:59.842654 7994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:12:59.842830 7994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:12:59.843091 7994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:12:59.843437 7994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:12:59.843498 7994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.843540 7994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:12:59.843568 7994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.849772 7994 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:37149
I20260925 10:12:59.849810 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:37149 every 8 connection(s)
I20260925 10:12:59.850128 7994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:12:59.858223 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 7994
I20260925 10:12:59.858347 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:12:59.858626 3683 external_mini_cluster.cc:1428] Setting key 36fc0019017015d2b6fdec13d00414f5
I20260925 10:12:59.860460 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:59.851119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42457 (local address 127.3.152.254:41025)
0925 10:12:59.851185 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:12:59.851187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:12:59.851476 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:12:59.852032 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:12:59.852034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:12:59.852045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:12:59.852112 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:12:59.853176 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.853670 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.854752 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.854933 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.854985 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:59.857396 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:12:59.857397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:12:59.857397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:12:59.857403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:12:59.858990 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.859413 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.859414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.859415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.859478 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:12:59.859757 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:12:59.859758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:12:59.859759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:12:59.859879 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:12:59.859954 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:12:59.860274 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 10:12:59.860367 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:12:59.860675 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:12:59.851075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41025 (local address 127.3.152.194:42457)
0925 10:12:59.851362 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 10:12:59.851395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 10:12:59.851899 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:12:59.852183 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:12:59.852186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:12:59.852487 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:12:59.853088 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:59.853094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.853777 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.853778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:59.854657 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:12:59.854660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:12:59.855016 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:12:59.855017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:12:59.855128 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:12:59.855792 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:12:59.855801 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:12:59.857309 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:12:59.859066 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:59.859073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:59.859076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:59.859322 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:59.859553 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:12:59.859554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:12:59.859554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:12:59.859665 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:12:59.859956 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:12:59.859986 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:12:59.860133 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:12:59.860252 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 10:12:59.860418 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260925 10:12:59.861370 8122 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41025
I20260925 10:12:59.861497 8122 heartbeater.cc:464] Registering TS with master...
I20260925 10:12:59.861671 8122 heartbeater.cc:511] Master 127.3.152.254:41025 requested a full tablet report, sending...
I20260925 10:12:59.862128 7797 ts_manager.cc:194] Registered new tserver with Master: aa5663677b584e9ea587c039335e8dfc (127.3.152.194:37149)
I20260925 10:12:59.862680 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:42457
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:12:59.887516 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:12:59.969251 8127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:12:59.969480 8127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:12:59.969506 8127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:12:59.970769 8127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:12:59.970822 8127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:12:59.970883 8127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:12:59.972525 8127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:12:59.972823 8127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:12:59.973068 8127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:12:59.978240 8132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.978258 8133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:12:59.978255 8135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:12:59.978729 8127 server_base.cc:1034] running on GCE node
I20260925 10:12:59.978976 8127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:12:59.979235 8127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:12:59.980378 8127 hybrid_clock.cc:648] HybridClock initialized: now 1790331179980355 us; error 34 us; skew 500 ppm
Sep 25 10:12:59 dist-test-slave-dzwq krb5kdc[7753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:12:59.983024 8127 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:12:59.983451 8127 fs_manager.cc:359] Metadata directory not provided
I20260925 10:12:59.983505 8127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:12:59.983687 8127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:12:59.984835 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "37bffaa411f74baba5fbc3d4e36a9043"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a0b6ad64ce0b41e5b390cd78c0c48d0a"
server_key_iv: "d1102b92a6fa876302a9df555196344f"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.985215 8127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "37bffaa411f74baba5fbc3d4e36a9043"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a0b6ad64ce0b41e5b390cd78c0c48d0a"
server_key_iv: "d1102b92a6fa876302a9df555196344f"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.986507 8127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:12:59.987353 8141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:12:59.987548 8127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:12:59.987730 8127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "37bffaa411f74baba5fbc3d4e36a9043"
format_stamp: "Formatted at 2026-09-25 10:12:59 on dist-test-slave-dzwq"
server_key: "a0b6ad64ce0b41e5b390cd78c0c48d0a"
server_key_iv: "d1102b92a6fa876302a9df555196344f"
server_key_version: "encryptionkey@0"
I20260925 10:12:59.987816 8127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:00.007475 8127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:00.010156 8127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:00.010378 8127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:00.010638 8127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:00.011000 8127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:00.011059 8127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.011106 8127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:00.011130 8127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.017421 8127 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:33301
I20260925 10:13:00.017457 8255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:33301 every 8 connection(s)
I20260925 10:13:00.017769 8127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:00.022586 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8127
I20260925 10:13:00.022689 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:00.022909 3683 external_mini_cluster.cc:1428] Setting key 8a9c874ee4216bcf99bae752eaeea720
Sep 25 10:13:00 dist-test-slave-dzwq krb5kdc[7753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331179, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:00.028656 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.018559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:38017 (local address 127.3.152.254:41025)
0925 10:13:00.018632 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.018634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.018869 (+ 235us) server_negotiation.cc:407] Connection header received
0925 10:13:00.019468 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.019469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.019482 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.019533 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:00.020626 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.021140 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.022206 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.022478 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.022556 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.025413 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:00.025414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:00.025415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:00.025423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:00.027115 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:00.027564 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:00.027565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:00.027565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:00.027682 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:00.027940 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:00.027941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:00.027941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:00.028120 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:00.028207 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.028499 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.028586 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:13:00.028878 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.018536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41025 (local address 127.3.152.195:38017)
0925 10:13:00.018734 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.018774 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.019351 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.019620 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.019631 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.019938 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:00.020509 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.020515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.021256 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.021258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.022102 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.022104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.022582 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.022583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.022746 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.023439 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:00.023476 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:00.025315 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:00.027224 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:00.027231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:00.027233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:00.027474 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:00.027768 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:00.027769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:00.027769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:00.027879 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:00.028209 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:00.028214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:00.028347 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.028441 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.028579 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 10:13:00.029429 8256 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41025
I20260925 10:13:00.029534 8256 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:00.029659 8256 heartbeater.cc:511] Master 127.3.152.254:41025 requested a full tablet report, sending...
I20260925 10:13:00.030061 7797 ts_manager.cc:194] Registered new tserver with Master: 37bffaa411f74baba5fbc3d4e36a9043 (127.3.152.195:33301)
I20260925 10:13:00.030421 7797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:38017
I20260925 10:13:00.035051 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:13:00.042915 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.036497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.3.152.254:41025)
0925 10:13:00.036580 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.036582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.036668 (+ 86us) server_negotiation.cc:407] Connection header received
0925 10:13:00.036788 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.036789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.036801 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.036860 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:00.037455 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.037949 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.038688 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.038903 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.038990 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.040310 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:00.040311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:00.040311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:00.040319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:00.041652 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:00.042014 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:00.042015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:00.042015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:00.042078 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:00.042321 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:00.042321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:00.042322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:00.042492 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:00.042602 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.042717 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.042805 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:13:00.044939 7797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:00.046624 7797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:00.053826 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:33301 (local address 127.0.0.1:56182)
0925 10:13:00.049739 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.049746 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.049774 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.050167 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.050168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.050170 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:00.050213 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.050215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051521 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.051522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.052606 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.052609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053504 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.053505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.053583 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.053596 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.053671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.053728 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":85,"thread_start_us":96,"threads_started":1}
I20260925 10:13:00.053856 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.3.152.195:33301)
0925 10:13:00.049910 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.049914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.049923 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:00.049986 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.049989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.050005 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.050083 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:00.050272 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051390 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.052697 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053400 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.053478 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.053536 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.053700 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.053761 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":75,"threads_started":1}
I20260925 10:13:00.053992 8267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:40303 (local address 127.0.0.1:40092)
0925 10:13:00.049492 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.049540 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.049626 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.050472 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.050473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.050475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:00.050528 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.050529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051459 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.051460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.052847 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.052848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053757 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.053759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.053840 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.053845 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.053878 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.053923 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":109,"thread_start_us":71,"threads_started":1}
I20260925 10:13:00.054111 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:37149 (local address 127.0.0.1:54108)
0925 10:13:00.049630 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.049641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.049694 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.050053 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.050057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.050059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:00.050146 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.050148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051496 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.051497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.052845 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.052848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053661 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.053662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.053714 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.053718 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.053761 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.053801 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260925 10:13:00.054216 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.3.152.194:37149)
0925 10:13:00.049811 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.049815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.049824 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:00.049858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.049860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.049896 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.049977 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:00.050221 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051397 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.052969 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053492 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.053560 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.053628 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.053866 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.054119 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260925 10:13:00.054833 8056 tablet_service.cc:1467] Processing CreateTablet for tablet d67f0fa34883408db965e84c5c5a18f1 (DEFAULT_TABLE table=test-table [id=53c5188f24354511a3e5bc9148841819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:00.054828 8189 tablet_service.cc:1467] Processing CreateTablet for tablet d67f0fa34883408db965e84c5c5a18f1 (DEFAULT_TABLE table=test-table [id=53c5188f24354511a3e5bc9148841819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:00.055199 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67f0fa34883408db965e84c5c5a18f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:00.055199 8189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67f0fa34883408db965e84c5c5a18f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:00.056252 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.049542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.3.152.193:40303)
0925 10:13:00.050338 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.050341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.050348 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:13:00.050377 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.050378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.050390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.050473 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:00.050679 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.051354 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.052933 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.053636 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.056058 (+ 2422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.056121 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.056146 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.056187 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":78,"threads_started":1}
I20260925 10:13:00.058669 8274 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Bootstrap starting.
I20260925 10:13:00.058749 8273 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Bootstrap starting.
I20260925 10:13:00.059340 8274 tablet_bootstrap.cc:654] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:00.059379 8273 tablet_bootstrap.cc:654] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:00.059285 7922 tablet_service.cc:1467] Processing CreateTablet for tablet d67f0fa34883408db965e84c5c5a18f1 (DEFAULT_TABLE table=test-table [id=53c5188f24354511a3e5bc9148841819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:00.059652 7922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67f0fa34883408db965e84c5c5a18f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:00.060009 8274 log.cc:824] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:00.060029 8273 log.cc:824] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:00.060956 8274 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: No bootstrap required, opened a new log
I20260925 10:13:00.061033 8273 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: No bootstrap required, opened a new log
I20260925 10:13:00.061041 8274 ts_tablet_manager.cc:1397] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:00.061100 8273 ts_tablet_manager.cc:1397] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:00.062516 8274 raft_consensus.cc:348] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.062685 8274 raft_consensus.cc:374] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:00.062739 8274 raft_consensus.cc:729] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa5663677b584e9ea587c039335e8dfc, State: Initialized, Role: FOLLOWER
I20260925 10:13:00.062858 8274 consensus_queue.cc:260] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.063195 8274 ts_tablet_manager.cc:1428] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:00.063197 8277 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Bootstrap starting.
I20260925 10:13:00.063325 8122 heartbeater.cc:503] Master 127.3.152.254:41025 was elected leader, sending a full tablet report...
I20260925 10:13:00.063210 8273 raft_consensus.cc:348] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.063367 8273 raft_consensus.cc:374] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:00.063418 8273 raft_consensus.cc:729] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37bffaa411f74baba5fbc3d4e36a9043, State: Initialized, Role: FOLLOWER
I20260925 10:13:00.063539 8273 consensus_queue.cc:260] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.063843 8273 ts_tablet_manager.cc:1428] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:13:00.063850 8256 heartbeater.cc:503] Master 127.3.152.254:41025 was elected leader, sending a full tablet report...
I20260925 10:13:00.064639 8277 tablet_bootstrap.cc:654] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:00.065243 8277 log.cc:824] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:00.065950 8277 tablet_bootstrap.cc:492] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: No bootstrap required, opened a new log
I20260925 10:13:00.066006 8277 ts_tablet_manager.cc:1397] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:13:00.067385 8277 raft_consensus.cc:348] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.067519 8277 raft_consensus.cc:374] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:00.067543 8277 raft_consensus.cc:729] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b44c7ea869894b27a4dfeef18945453a, State: Initialized, Role: FOLLOWER
I20260925 10:13:00.067684 8277 consensus_queue.cc:260] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.068034 8277 ts_tablet_manager.cc:1428] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:00.068046 7989 heartbeater.cc:503] Master 127.3.152.254:41025 was elected leader, sending a full tablet report...
W20260925 10:13:00.100706 8123 tablet.cc:2369] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:13:00.188243 7990 tablet.cc:2369] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:13:00.268345 8257 tablet.cc:2369] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:13:00.395776 8279 raft_consensus.cc:482] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:00.395910 8279 raft_consensus.cc:504] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.396493 8279 leader_election.cc:288] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa5663677b584e9ea587c039335e8dfc (127.3.152.194:37149), b44c7ea869894b27a4dfeef18945453a (127.3.152.193:40303)
I20260925 10:13:00.399566 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.396817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:58087 (local address 127.3.152.194:37149)
0925 10:13:00.396860 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.396863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.396875 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:00.396964 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.396965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.396973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.397051 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:00.397357 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.397896 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.398850 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.399213 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.399299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.399309 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.399460 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.399494 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:00.399576 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.396759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:37149 (local address 127.3.152.195:58087)
0925 10:13:00.396824 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.396838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.396888 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.397058 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.397059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.397061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:00.397136 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.397138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.397984 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.397985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.398753 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.398755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.399298 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.399299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.399385 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.399387 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.399454 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.399494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:13:00.399631 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.396805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:37163 (local address 127.3.152.193:40303)
0925 10:13:00.396851 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.396853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.397210 (+ 357us) server_negotiation.cc:407] Connection header received
0925 10:13:00.397242 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.397243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.397255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.397315 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:00.397473 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.397994 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.398918 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.399279 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.399363 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.399370 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.399523 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.399558 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:00.399904 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.396762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:40303 (local address 127.3.152.195:37163)
0925 10:13:00.397122 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.397131 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.397190 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.397330 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.397330 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.397340 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:00.397394 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.397395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.398085 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.398086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.398817 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:00.398819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.399362 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.399363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:00.399434 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.399436 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.399513 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.399550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":184,"thread_start_us":122,"threads_started":1}
I20260925 10:13:00.400012 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67f0fa34883408db965e84c5c5a18f1" candidate_uuid: "37bffaa411f74baba5fbc3d4e36a9043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5663677b584e9ea587c039335e8dfc" is_pre_election: true
I20260925 10:13:00.400207 8076 raft_consensus.cc:2454] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bffaa411f74baba5fbc3d4e36a9043 in term 0.
I20260925 10:13:00.400277 7942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67f0fa34883408db965e84c5c5a18f1" candidate_uuid: "37bffaa411f74baba5fbc3d4e36a9043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44c7ea869894b27a4dfeef18945453a" is_pre_election: true
I20260925 10:13:00.400457 7942 raft_consensus.cc:2454] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37bffaa411f74baba5fbc3d4e36a9043 in term 0.
I20260925 10:13:00.400483 8142 leader_election.cc:302] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bffaa411f74baba5fbc3d4e36a9043, aa5663677b584e9ea587c039335e8dfc; no voters:
I20260925 10:13:00.400619 8279 raft_consensus.cc:2781] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:00.400681 8279 raft_consensus.cc:482] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:00.400707 8279 raft_consensus.cc:3037] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:00.401423 8279 raft_consensus.cc:504] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.401578 8279 leader_election.cc:288] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [CANDIDATE]: Term 1 election: Requested vote from peers aa5663677b584e9ea587c039335e8dfc (127.3.152.194:37149), b44c7ea869894b27a4dfeef18945453a (127.3.152.193:40303)
I20260925 10:13:00.401747 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67f0fa34883408db965e84c5c5a18f1" candidate_uuid: "37bffaa411f74baba5fbc3d4e36a9043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa5663677b584e9ea587c039335e8dfc"
I20260925 10:13:00.401840 7942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67f0fa34883408db965e84c5c5a18f1" candidate_uuid: "37bffaa411f74baba5fbc3d4e36a9043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b44c7ea869894b27a4dfeef18945453a"
I20260925 10:13:00.401861 8076 raft_consensus.cc:3037] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:00.401932 7942 raft_consensus.cc:3037] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:00.402896 7942 raft_consensus.cc:2454] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bffaa411f74baba5fbc3d4e36a9043 in term 1.
I20260925 10:13:00.403002 8076 raft_consensus.cc:2454] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37bffaa411f74baba5fbc3d4e36a9043 in term 1.
I20260925 10:13:00.403070 8143 leader_election.cc:302] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37bffaa411f74baba5fbc3d4e36a9043, b44c7ea869894b27a4dfeef18945453a; no voters:
I20260925 10:13:00.403177 8279 raft_consensus.cc:2781] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:00.403363 8279 raft_consensus.cc:686] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 1 LEADER]: Becoming Leader. State: Replica: 37bffaa411f74baba5fbc3d4e36a9043, State: Running, Role: LEADER
I20260925 10:13:00.403462 8279 consensus_queue.cc:237] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } }
I20260925 10:13:00.404326 7797 catalog_manager.cc:5495] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37bffaa411f74baba5fbc3d4e36a9043 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "37bffaa411f74baba5fbc3d4e36a9043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37bffaa411f74baba5fbc3d4e36a9043" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 } health_report { overall_health: UNKNOWN } } }
I20260925 10:13:00.495122 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.492498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.3.152.195:33301)
0925 10:13:00.492571 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.492573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.492582 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:00.492622 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.492623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.492649 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.492728 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:00.492885 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.493609 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.494396 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:00.494602 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:00.494672 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:00.494836 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:00.494937 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:00.495011 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.495025 (+ 14us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.495044 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:13:00.497107 7942 raft_consensus.cc:1261] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 1 FOLLOWER]: Refusing update from remote peer 37bffaa411f74baba5fbc3d4e36a9043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:00.497107 8076 raft_consensus.cc:1261] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 1 FOLLOWER]: Refusing update from remote peer 37bffaa411f74baba5fbc3d4e36a9043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:00.497480 8283 consensus_queue.cc:1035] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa5663677b584e9ea587c039335e8dfc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:00.497622 8279 consensus_queue.cc:1035] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b44c7ea869894b27a4dfeef18945453a" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:00.500062 8288 mvcc.cc:205] Tried to move back new op lower bound from 7333196515312992256 to 7333196514933280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333196515312992256}]
I20260925 10:13:00.500067 8286 mvcc.cc:205] Tried to move back new op lower bound from 7333196515312992256 to 7333196514933280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333196515312992256}]
I20260925 10:13:00.500761 8287 mvcc.cc:205] Tried to move back new op lower bound from 7333196515312992256 to 7333196514933280768. Current Snapshot: MvccSnapshot[applied={T|T < 7333196515312992256}]
I20260925 10:13:00.502388 7796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:00.502507 7796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:00.503334 7796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcbd98b4ed30484181822759383f342d: Sending DeleteTablet for 3 replicas of tablet d67f0fa34883408db965e84c5c5a18f1
I20260925 10:13:00.503682 7922 tablet_service.cc:1514] Processing DeleteTablet for tablet d67f0fa34883408db965e84c5c5a18f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:00 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40092
I20260925 10:13:00.503669 8189 tablet_service.cc:1514] Processing DeleteTablet for tablet d67f0fa34883408db965e84c5c5a18f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:00 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:56182
I20260925 10:13:00.503695 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet d67f0fa34883408db965e84c5c5a18f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:00 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:54108
I20260925 10:13:00.503914 8296 tablet_replica.cc:330] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: stopping tablet replica
I20260925 10:13:00.504029 8296 raft_consensus.cc:2229] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:00.504189 8296 raft_consensus.cc:2258] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:00.504431 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7860
I20260925 10:13:00.504534 8295 tablet_replica.cc:330] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: stopping tablet replica
I20260925 10:13:00.504621 8295 raft_consensus.cc:2229] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:00.504614 8296 ts_tablet_manager.cc:1905] T d67f0fa34883408db965e84c5c5a18f1 P b44c7ea869894b27a4dfeef18945453a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:00.504704 8295 raft_consensus.cc:2258] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:00.505041 8297 tablet_replica.cc:330] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: stopping tablet replica
I20260925 10:13:00.505151 8297 raft_consensus.cc:2229] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:00.505254 8297 raft_consensus.cc:2258] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:00.506237 8295 ts_tablet_manager.cc:1905] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:00.506493 8297 ts_tablet_manager.cc:1905] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:00.508391 8297 ts_tablet_manager.cc:1918] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:00.508457 8297 log.cc:1196] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/d67f0fa34883408db965e84c5c5a18f1
I20260925 10:13:00.508720 8297 ts_tablet_manager.cc:1939] T d67f0fa34883408db965e84c5c5a18f1 P aa5663677b584e9ea587c039335e8dfc: Deleting consensus metadata
I20260925 10:13:00.509366 7781 catalog_manager.cc:4841] TS aa5663677b584e9ea587c039335e8dfc (127.3.152.194:37149): tablet d67f0fa34883408db965e84c5c5a18f1 (table test-table [id=53c5188f24354511a3e5bc9148841819]) successfully deleted
W20260925 10:13:00.511816 7782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:40303 (error 108)
I20260925 10:13:00.512109 8295 ts_tablet_manager.cc:1918] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:00.512367 8269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.512235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:40303 (local address 127.0.0.1:40108)
0925 10:13:00.512281 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.512322 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:40303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 10:13:00.512492 7782 catalog_manager.cc:4584] TS b44c7ea869894b27a4dfeef18945453a (127.3.152.193:40303): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d67f0fa34883408db965e84c5c5a18f1: Network error: Client connection negotiation failed: client connection to 127.3.152.193:40303: connect: Connection refused (error 111)
I20260925 10:13:00.512507 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7994
I20260925 10:13:00.517102 8295 log.cc:1196] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/d67f0fa34883408db965e84c5c5a18f1
I20260925 10:13:00.517428 8295 ts_tablet_manager.cc:1939] T d67f0fa34883408db965e84c5c5a18f1 P 37bffaa411f74baba5fbc3d4e36a9043: Deleting consensus metadata
I20260925 10:13:00.517868 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8127
I20260925 10:13:00.517885 7783 catalog_manager.cc:4841] TS 37bffaa411f74baba5fbc3d4e36a9043 (127.3.152.195:33301): tablet d67f0fa34883408db965e84c5c5a18f1 (table test-table [id=53c5188f24354511a3e5bc9148841819]) successfully deleted
I20260925 10:13:00.522827 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 7766
2026-09-25T10:13:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2329 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T10:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:00Z Disabled control of system clock
I20260925 10:13:00.550321 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:44367
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:13:00.636821 8302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:00.637037 8302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:00.637058 8302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:00.638166 8302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:00.638202 8302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:00.638216 8302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:00.638227 8302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:00.638239 8302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:00.639568 8302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:44367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:00.639779 8302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:00.640031 8302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:00.645080 8308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.645080 8310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.645160 8307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:00.645226 8302 server_base.cc:1034] running on GCE node
I20260925 10:13:00.645524 8302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:00.645834 8302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:00.647039 8302 hybrid_clock.cc:648] HybridClock initialized: now 1790331180647024 us; error 34 us; skew 500 ppm
I20260925 10:13:00.648499 8302 webserver.cc:460] Webserver started at http://127.3.152.254:35721/ using document root <none> and password file <none>
I20260925 10:13:00.648797 8302 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:00.648869 8302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:00.649016 8302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:00.650151 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "e20d04f478c04617ae7bfcc8fb52bac9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "1f743240c93e99620b9889eca9a5bf9d"
server_key_iv: "2a280c644c29c477acd29888546426d5"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.650560 8302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "e20d04f478c04617ae7bfcc8fb52bac9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "1f743240c93e99620b9889eca9a5bf9d"
server_key_iv: "2a280c644c29c477acd29888546426d5"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.651880 8302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:13:00.652657 8316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.652851 8302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:00.652956 8302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "e20d04f478c04617ae7bfcc8fb52bac9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "1f743240c93e99620b9889eca9a5bf9d"
server_key_iv: "2a280c644c29c477acd29888546426d5"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.653030 8302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:00.656540 8302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:00.656813 8302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:00.656955 8302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:00.660836 8302 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:44367
I20260925 10:13:00.660866 8369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:44367 every 8 connection(s)
I20260925 10:13:00.661193 8302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:00.661836 8370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:00.664656 8370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: Bootstrap starting.
I20260925 10:13:00.664849 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8302
I20260925 10:13:00.664966 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:00.665148 3683 external_mini_cluster.cc:1428] Setting key 355e186ae314b34821b2a3c6838f95b7
I20260925 10:13:00.665340 8370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:00.665693 8370 log.cc:824] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:00.666409 8370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: No bootstrap required, opened a new log
I20260925 10:13:00.667820 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.665699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.3.152.254:44367)
0925 10:13:00.665923 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.665934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.665969 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:13:00.666504 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.666513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.666578 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.666696 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:00.667356 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:00.667359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:00.667360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:00.667364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:00.667402 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:00.667411 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:00.667479 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.667559 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.667629 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260925 10:13:00.668421 8370 raft_consensus.cc:348] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } }
I20260925 10:13:00.668597 8370 raft_consensus.cc:374] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:00.668650 8370 raft_consensus.cc:729] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20d04f478c04617ae7bfcc8fb52bac9, State: Initialized, Role: FOLLOWER
I20260925 10:13:00.668776 8370 consensus_queue.cc:260] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } }
I20260925 10:13:00.668857 8370 raft_consensus.cc:388] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:00.668880 8370 raft_consensus.cc:482] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:00.668946 8370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:00.669735 8370 raft_consensus.cc:504] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } }
I20260925 10:13:00.669849 8370 leader_election.cc:302] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e20d04f478c04617ae7bfcc8fb52bac9; no voters:
I20260925 10:13:00.669997 8370 leader_election.cc:288] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:00.670065 8375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:00.670212 8375 raft_consensus.cc:686] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [term 1 LEADER]: Becoming Leader. State: Replica: e20d04f478c04617ae7bfcc8fb52bac9, State: Running, Role: LEADER
I20260925 10:13:00.670274 8370 sys_catalog.cc:564] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:00.670307 8375 consensus_queue.cc:237] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } }
I20260925 10:13:00.670639 8377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e20d04f478c04617ae7bfcc8fb52bac9. Latest consensus state: current_term: 1 leader_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } } }
I20260925 10:13:00.670670 8376 sys_catalog.cc:455] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e20d04f478c04617ae7bfcc8fb52bac9" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44367 } } }
I20260925 10:13:00.670734 8377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:00.670746 8376 sys_catalog.cc:458] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:00.670955 8382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:00.671423 8382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:00.673004 8382 catalog_manager.cc:1347] Generated new cluster ID: 3cd0d42547cd43a3ad62bd49c4adc274
I20260925 10:13:00.673046 8382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:00.692260 8382 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:00.692473 8382 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:00.699913 8382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: Generated new TSK 0
I20260925 10:13:00.700168 8382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:13:00.710577 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--builtin_ntp_servers=127.3.152.212:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:13:00.793881 8394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:00.794101 8394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:00.794126 8394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:00.795334 8394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:00.795375 8394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:00.795401 8394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:00.796797 8394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:00.796995 8394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:00.797242 8394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:00.802757 8402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.802841 8399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.802841 8400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:00.803058 8394 server_base.cc:1034] running on GCE node
I20260925 10:13:00.803261 8394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:00.803494 8394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:00.804741 8394 hybrid_clock.cc:648] HybridClock initialized: now 1790331180804714 us; error 38 us; skew 500 ppm
I20260925 10:13:00.806156 8394 webserver.cc:460] Webserver started at http://127.3.152.193:38013/ using document root <none> and password file <none>
I20260925 10:13:00.806433 8394 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:00.806480 8394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:00.806659 8394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:00.807842 8394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "e58cea93d60244a18a3dde5cfedc48f9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "55acf72c99f61f107747c30d53b657a7"
server_key_iv: "2aa37e01d26186e6962c4c8d638d01c7"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.808269 8394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "e58cea93d60244a18a3dde5cfedc48f9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "55acf72c99f61f107747c30d53b657a7"
server_key_iv: "2aa37e01d26186e6962c4c8d638d01c7"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.809686 8394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:00.810465 8408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.810796 8394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:00.810907 8394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "e58cea93d60244a18a3dde5cfedc48f9"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "55acf72c99f61f107747c30d53b657a7"
server_key_iv: "2aa37e01d26186e6962c4c8d638d01c7"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.810997 8394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:00.814872 8394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:00.815193 8394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:00.815378 8394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:00.815621 8394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:00.816002 8394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:00.816076 8394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.816138 8394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:00.816169 8394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.822335 8394 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:36559
I20260925 10:13:00.822371 8523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:36559 every 8 connection(s)
I20260925 10:13:00.822758 8394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:00.825640 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8394
I20260925 10:13:00.825728 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:00.825922 3683 external_mini_cluster.cc:1428] Setting key 7f86dd06b3dc353a5d6de927799c7d8d
I20260925 10:13:00.826267 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.823641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:58939 (local address 127.3.152.254:44367)
0925 10:13:00.823743 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.823746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.823991 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:13:00.824576 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.824577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.824586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.824636 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:00.825952 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:00.825954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:00.825954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:00.825959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:00.825976 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:00.825980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:00.826038 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.826173 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.826189 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:13:00.826493 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.823614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44367 (local address 127.3.152.193:58939)
0925 10:13:00.823837 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.823873 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.824443 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.824733 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.824737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.824746 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:00.825780 (+ 1034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:00.825781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:00.825820 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:00.825852 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:00.826064 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:00.826071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:00.826074 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.826185 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.826253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260925 10:13:00.827033 8524 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44367
I20260925 10:13:00.827143 8524 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:00.827211 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--builtin_ntp_servers=127.3.152.212:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 10:13:00.827330 8524 heartbeater.cc:511] Master 127.3.152.254:44367 requested a full tablet report, sending...
I20260925 10:13:00.827899 8334 ts_manager.cc:194] Registered new tserver with Master: e58cea93d60244a18a3dde5cfedc48f9 (127.3.152.193:36559)
W20260925 10:13:00.911381 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:00.911626 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:00.911666 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:00.912981 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:00.913043 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:00.913092 8527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:00.914680 8527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:00.914955 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:00.915270 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:00.920317 8533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.920364 8532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:00.920521 8535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:00.920414 8527 server_base.cc:1034] running on GCE node
I20260925 10:13:00.920765 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:00.921015 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:00.922211 8527 hybrid_clock.cc:648] HybridClock initialized: now 1790331180922198 us; error 44 us; skew 500 ppm
I20260925 10:13:00.923454 8527 webserver.cc:460] Webserver started at http://127.3.152.194:43783/ using document root <none> and password file <none>
I20260925 10:13:00.923699 8527 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:00.923770 8527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:00.923924 8527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:00.925016 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "24e4a7268398488292ec89004acf1c3f"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "a4cc84bf0a6f4df559b4780591ffa6cb"
server_key_iv: "dcbfd107490293b272a5425d43f86dce"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.925397 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "24e4a7268398488292ec89004acf1c3f"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "a4cc84bf0a6f4df559b4780591ffa6cb"
server_key_iv: "dcbfd107490293b272a5425d43f86dce"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.926754 8527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:00.927451 8541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.927623 8527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:00.927723 8527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "24e4a7268398488292ec89004acf1c3f"
format_stamp: "Formatted at 2026-09-25 10:13:00 on dist-test-slave-dzwq"
server_key: "a4cc84bf0a6f4df559b4780591ffa6cb"
server_key_iv: "dcbfd107490293b272a5425d43f86dce"
server_key_version: "encryptionkey@0"
I20260925 10:13:00.927807 8527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:00.931124 8527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:00.931385 8527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:00.931523 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:00.931731 8527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:00.932101 8527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:00.932159 8527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.932197 8527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:00.932224 8527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:00.937956 8527 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:44891
I20260925 10:13:00.938000 8655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:44891 every 8 connection(s)
I20260925 10:13:00.938366 8527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:00.941440 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.939159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42793 (local address 127.3.152.254:44367)
0925 10:13:00.939230 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:00.939232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:00.939435 (+ 203us) server_negotiation.cc:407] Connection header received
0925 10:13:00.939876 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:00.939877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:00.939884 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:00.939946 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:00.941150 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:00.941151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:00.941151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:00.941156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:00.941171 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:00.941176 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:00.941234 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:00.941372 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 10:13:00.941386 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:13:00.941640 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:00.939084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44367 (local address 127.3.152.194:42793)
0925 10:13:00.939305 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:00.939345 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:00.939796 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:00.940055 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:00.940066 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:00.940074 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:00.940986 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:00.940987 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:00.941015 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:00.941045 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:00.941261 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:00.941265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:00.941268 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:00.941329 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:13:00.941378 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 10:13:00.942178 8656 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44367
I20260925 10:13:00.942291 8656 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:00.942358 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8527
I20260925 10:13:00.942461 8656 heartbeater.cc:511] Master 127.3.152.254:44367 requested a full tablet report, sending...
I20260925 10:13:00.942466 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:00.942683 3683 external_mini_cluster.cc:1428] Setting key 8ee6ae95204567df739e522fbbd58ce1
I20260925 10:13:00.942914 8334 ts_manager.cc:194] Registered new tserver with Master: 24e4a7268398488292ec89004acf1c3f (127.3.152.194:44891)
I20260925 10:13:00.943706 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--builtin_ntp_servers=127.3.152.212:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 10:13:01.024538 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:01.024746 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:01.024768 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:01.025940 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:01.025981 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:01.026006 8659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:01.027390 8659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:01.027637 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:01.027932 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:01.033070 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.033109 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.033098 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.033258 8659 server_base.cc:1034] running on GCE node
I20260925 10:13:01.033488 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:01.033706 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:01.034845 8659 hybrid_clock.cc:648] HybridClock initialized: now 1790331181034834 us; error 27 us; skew 500 ppm
I20260925 10:13:01.036253 8659 webserver.cc:460] Webserver started at http://127.3.152.195:38611/ using document root <none> and password file <none>
I20260925 10:13:01.036497 8659 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:01.036558 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:01.036794 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:01.037904 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "c7452445fc434f149ec3850072775f33"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "cea67a5ce5ab0ea1a7bc634974170080"
server_key_iv: "c4c25aba6b076ee3806a225e849da411"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.038297 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "c7452445fc434f149ec3850072775f33"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "cea67a5ce5ab0ea1a7bc634974170080"
server_key_iv: "c4c25aba6b076ee3806a225e849da411"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.039737 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:01.040650 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.040804 8659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:01.040890 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "c7452445fc434f149ec3850072775f33"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "cea67a5ce5ab0ea1a7bc634974170080"
server_key_iv: "c4c25aba6b076ee3806a225e849da411"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.040954 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:01.044375 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:01.044608 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:01.044703 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:01.044859 8659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:01.045094 8659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:01.045122 8659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.045143 8659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:01.045156 8659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.050781 8659 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:40775
I20260925 10:13:01.050863 8786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:40775 every 8 connection(s)
I20260925 10:13:01.051183 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:01.054392 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.052042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:41677 (local address 127.3.152.254:44367)
0925 10:13:01.052122 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.052125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.052328 (+ 203us) server_negotiation.cc:407] Connection header received
0925 10:13:01.052870 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.052871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.052879 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.052937 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.054115 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.054116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.054117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.054121 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.054138 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.054142 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.054198 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.054303 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.054318 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:13:01.054589 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.051978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44367 (local address 127.3.152.195:41677)
0925 10:13:01.052202 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.052240 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.052754 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.053013 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.053017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.053027 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.053930 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.053931 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.053959 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.053987 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.054210 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.054214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.054217 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.054312 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.054374 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260925 10:13:01.055114 8787 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44367
I20260925 10:13:01.055212 8787 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:01.055362 8787 heartbeater.cc:511] Master 127.3.152.254:44367 requested a full tablet report, sending...
I20260925 10:13:01.055820 8333 ts_manager.cc:194] Registered new tserver with Master: c7452445fc434f149ec3850072775f33 (127.3.152.195:40775)
I20260925 10:13:01.058626 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8659
I20260925 10:13:01.058760 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:01.058977 3683 external_mini_cluster.cc:1428] Setting key e48c5076cf81248b8d9649635e3d2aaa
I20260925 10:13:01.059890 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:13:01.062539 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.061416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.3.152.254:44367)
0925 10:13:01.061474 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.061475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.061485 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:01.061512 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.061512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.061524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.061573 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.062365 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.062366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.062366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.062369 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.062382 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.062386 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.062431 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.062477 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.062488 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:13:01.064239 8333 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:01.065846 8333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:01.070384 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:44891 (local address 127.0.0.1:60802)
0925 10:13:01.068584 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.068664 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.068704 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.069197 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.069205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.069212 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.069927 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.069927 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.069939 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.069964 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.070208 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.070215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.070219 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.070297 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.070311 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":77,"threads_started":1}
I20260925 10:13:01.070469 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.3.152.194:44891)
0925 10:13:01.068709 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.068713 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.068724 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:01.068757 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.068759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.068767 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.068848 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.070041 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.070052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.070052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.070057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.070106 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.070137 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.070202 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.070326 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.070343 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260925 10:13:01.070698 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.3.152.193:36559)
0925 10:13:01.068413 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.068418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.068534 (+ 116us) server_negotiation.cc:407] Connection header received
0925 10:13:01.068836 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.068843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.068854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.068909 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.070082 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.070089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.070089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.070093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.070145 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.070164 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.070243 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.070329 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.070349 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260925 10:13:01.070991 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.3.152.195:40775)
0925 10:13:01.068498 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.068711 (+ 213us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.068724 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:13:01.068753 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.068756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.068767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.068848 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.070241 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.070249 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.070249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.070254 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.070285 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.070310 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.070365 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.070898 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.070914 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":91,"threads_started":1}
I20260925 10:13:01.070508 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:40775 (local address 127.0.0.1:57138)
0925 10:13:01.068483 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.068493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.068540 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.069197 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.069205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.069212 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.069920 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.069921 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.069939 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.069964 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.070389 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.070389 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.070390 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.070433 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.070446 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":4,"thread_start_us":66,"threads_started":1}
I20260925 10:13:01.071188 8795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.068107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:36559 (local address 127.0.0.1:54066)
0925 10:13:01.068393 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.068461 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.068756 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.069249 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.069250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.069252 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.069989 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.069989 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.069997 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.070014 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.070237 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.070238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.070238 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.070820 (+ 582us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.070836 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":78,"threads_started":1}
I20260925 10:13:01.071328 8590 tablet_service.cc:1467] Processing CreateTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 (DEFAULT_TABLE table=test-table [id=8713beff6e1f41fea8b10d4b6eb1f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:01.071743 8590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.071775 8721 tablet_service.cc:1467] Processing CreateTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 (DEFAULT_TABLE table=test-table [id=8713beff6e1f41fea8b10d4b6eb1f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:01.071866 8458 tablet_service.cc:1467] Processing CreateTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 (DEFAULT_TABLE table=test-table [id=8713beff6e1f41fea8b10d4b6eb1f08e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:01.072171 8721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.072180 8458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.074476 8801 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Bootstrap starting.
I20260925 10:13:01.075093 8801 tablet_bootstrap.cc:654] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.075475 8801 log.cc:824] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.075577 8802 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: Bootstrap starting.
I20260925 10:13:01.076156 8803 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Bootstrap starting.
I20260925 10:13:01.076215 8801 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: No bootstrap required, opened a new log
I20260925 10:13:01.076246 8802 tablet_bootstrap.cc:654] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.076303 8801 ts_tablet_manager.cc:1397] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:01.076582 8802 log.cc:824] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.076869 8803 tablet_bootstrap.cc:654] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.077220 8803 log.cc:824] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.077437 8802 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: No bootstrap required, opened a new log
I20260925 10:13:01.077505 8802 ts_tablet_manager.cc:1397] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:01.077989 8803 tablet_bootstrap.cc:492] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: No bootstrap required, opened a new log
I20260925 10:13:01.078065 8803 ts_tablet_manager.cc:1397] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:01.078791 8801 raft_consensus.cc:348] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.078948 8801 raft_consensus.cc:374] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.078990 8801 raft_consensus.cc:729] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e4a7268398488292ec89004acf1c3f, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.079097 8801 consensus_queue.cc:260] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.079351 8801 ts_tablet_manager.cc:1428] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:13:01.079481 8656 heartbeater.cc:503] Master 127.3.152.254:44367 was elected leader, sending a full tablet report...
I20260925 10:13:01.080209 8802 raft_consensus.cc:348] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.080415 8802 raft_consensus.cc:374] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.080458 8802 raft_consensus.cc:729] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58cea93d60244a18a3dde5cfedc48f9, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.080605 8802 consensus_queue.cc:260] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.080729 8803 raft_consensus.cc:348] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.080881 8802 ts_tablet_manager.cc:1428] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:13:01.080884 8803 raft_consensus.cc:374] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.080921 8803 raft_consensus.cc:729] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7452445fc434f149ec3850072775f33, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.080937 8524 heartbeater.cc:503] Master 127.3.152.254:44367 was elected leader, sending a full tablet report...
I20260925 10:13:01.081022 8803 consensus_queue.cc:260] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.081277 8803 ts_tablet_manager.cc:1428] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:13:01.081342 8787 heartbeater.cc:503] Master 127.3.152.254:44367 was elected leader, sending a full tablet report...
I20260925 10:13:01.094990 8808 raft_consensus.cc:482] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:01.095124 8808 raft_consensus.cc:504] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.095434 8808 leader_election.cc:288] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7452445fc434f149ec3850072775f33 (127.3.152.195:40775), 24e4a7268398488292ec89004acf1c3f (127.3.152.194:44891)
I20260925 10:13:01.096889 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.095692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:40775 (local address 127.3.152.193:42401)
0925 10:13:01.095764 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.095778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.095834 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.096022 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.096023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.096026 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.096529 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.096529 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.096537 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.096548 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.096742 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.096743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.096744 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.096800 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.096815 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 10:13:01.096889 8798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.095800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42401 (local address 127.3.152.195:40775)
0925 10:13:01.095874 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.095876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.095885 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:01.095912 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.095913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.095919 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.096010 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.096646 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.096648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.096648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.096653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.096676 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.096680 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.096752 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.096802 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.096816 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:13:01.097105 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.095994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:59281 (local address 127.3.152.194:44891)
0925 10:13:01.096040 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.096042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.096154 (+ 112us) server_negotiation.cc:407] Connection header received
0925 10:13:01.096222 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.096223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.096228 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.096289 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.096810 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.096812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.096812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.096817 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.096833 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.096837 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.096911 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.097020 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.097037 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:13:01.097114 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.095845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:44891 (local address 127.3.152.193:59281)
0925 10:13:01.096077 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.096088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.096144 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.096315 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.096316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.096317 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:01.096689 (+ 372us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:01.096689 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:01.096696 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:01.096707 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:01.096909 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:01.096910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:01.096910 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 10:13:01.097012 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 10:13:01.097029 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":154,"threads_started":1}
I20260925 10:13:01.097254 8741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743aeb5d446947c3a5a3d0c9fa3b74c3" candidate_uuid: "e58cea93d60244a18a3dde5cfedc48f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7452445fc434f149ec3850072775f33" is_pre_election: true
I20260925 10:13:01.097394 8741 raft_consensus.cc:2454] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58cea93d60244a18a3dde5cfedc48f9 in term 0.
I20260925 10:13:01.097648 8412 leader_election.cc:302] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7452445fc434f149ec3850072775f33, e58cea93d60244a18a3dde5cfedc48f9; no voters:
I20260925 10:13:01.097765 8610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743aeb5d446947c3a5a3d0c9fa3b74c3" candidate_uuid: "e58cea93d60244a18a3dde5cfedc48f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e4a7268398488292ec89004acf1c3f" is_pre_election: true
I20260925 10:13:01.097936 8610 raft_consensus.cc:2454] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58cea93d60244a18a3dde5cfedc48f9 in term 0.
I20260925 10:13:01.098079 8808 raft_consensus.cc:2781] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:01.098157 8808 raft_consensus.cc:482] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:01.098191 8808 raft_consensus.cc:3037] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.098914 8808 raft_consensus.cc:504] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.099057 8808 leader_election.cc:288] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [CANDIDATE]: Term 1 election: Requested vote from peers c7452445fc434f149ec3850072775f33 (127.3.152.195:40775), 24e4a7268398488292ec89004acf1c3f (127.3.152.194:44891)
I20260925 10:13:01.099220 8741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743aeb5d446947c3a5a3d0c9fa3b74c3" candidate_uuid: "e58cea93d60244a18a3dde5cfedc48f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7452445fc434f149ec3850072775f33"
I20260925 10:13:01.099220 8610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743aeb5d446947c3a5a3d0c9fa3b74c3" candidate_uuid: "e58cea93d60244a18a3dde5cfedc48f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24e4a7268398488292ec89004acf1c3f"
I20260925 10:13:01.099301 8610 raft_consensus.cc:3037] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.099301 8741 raft_consensus.cc:3037] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.100222 8741 raft_consensus.cc:2454] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58cea93d60244a18a3dde5cfedc48f9 in term 1.
I20260925 10:13:01.100222 8610 raft_consensus.cc:2454] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58cea93d60244a18a3dde5cfedc48f9 in term 1.
I20260925 10:13:01.100400 8412 leader_election.cc:302] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7452445fc434f149ec3850072775f33, e58cea93d60244a18a3dde5cfedc48f9; no voters:
I20260925 10:13:01.100512 8808 raft_consensus.cc:2781] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:01.100666 8808 raft_consensus.cc:686] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [term 1 LEADER]: Becoming Leader. State: Replica: e58cea93d60244a18a3dde5cfedc48f9, State: Running, Role: LEADER
I20260925 10:13:01.100777 8808 consensus_queue.cc:237] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } }
I20260925 10:13:01.101657 8332 catalog_manager.cc:5495] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e58cea93d60244a18a3dde5cfedc48f9 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "e58cea93d60244a18a3dde5cfedc48f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58cea93d60244a18a3dde5cfedc48f9" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 36559 } health_report { overall_health: HEALTHY } } }
I20260925 10:13:01.109356 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.108268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.3.152.193:36559)
0925 10:13:01.108323 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.108325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.108334 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:01.108383 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.108384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.108398 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.108464 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.109080 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.109082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.109082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.109088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.109106 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.109110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.109193 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.109257 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.109273 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:13:01.111542 8610 raft_consensus.cc:1261] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 1 FOLLOWER]: Refusing update from remote peer e58cea93d60244a18a3dde5cfedc48f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:01.111533 8741 raft_consensus.cc:1261] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 1 FOLLOWER]: Refusing update from remote peer e58cea93d60244a18a3dde5cfedc48f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:01.111936 8808 consensus_queue.cc:1035] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7452445fc434f149ec3850072775f33" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:01.112134 8811 consensus_queue.cc:1035] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P e58cea93d60244a18a3dde5cfedc48f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24e4a7268398488292ec89004acf1c3f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:01.114670 8814 mvcc.cc:205] Tried to move back new op lower bound from 7333196517829263360 to 7333196517789405184. Current Snapshot: MvccSnapshot[applied={T|T < 7333196517829263360}]
I20260925 10:13:01.115586 8815 mvcc.cc:205] Tried to move back new op lower bound from 7333196517829263360 to 7333196517789405184. Current Snapshot: MvccSnapshot[applied={T|T < 7333196517829263360}]
I20260925 10:13:01.117552 8332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:01.117663 8332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:01.118533 8332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e20d04f478c04617ae7bfcc8fb52bac9: Sending DeleteTablet for 3 replicas of tablet 743aeb5d446947c3a5a3d0c9fa3b74c3
I20260925 10:13:01.118867 8721 tablet_service.cc:1514] Processing DeleteTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:01 UTC) from {username='slave'} at 127.0.0.1:57138
I20260925 10:13:01.118871 8458 tablet_service.cc:1514] Processing DeleteTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:01 UTC) from {username='slave'} at 127.0.0.1:54066
I20260925 10:13:01.118929 8590 tablet_service.cc:1514] Processing DeleteTablet for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:01 UTC) from {username='slave'} at 127.0.0.1:60802
I20260925 10:13:01.119166 8824 tablet_replica.cc:330] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: stopping tablet replica
I20260925 10:13:01.119297 8824 raft_consensus.cc:2229] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:01.119372 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8394
I20260925 10:13:01.119988 8824 raft_consensus.cc:2258] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:01.120711 8824 ts_tablet_manager.cc:1905] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:01.121017 8826 tablet_replica.cc:330] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: stopping tablet replica
I20260925 10:13:01.121129 8826 raft_consensus.cc:2229] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:01.121229 8826 raft_consensus.cc:2258] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:01.121688 8826 ts_tablet_manager.cc:1905] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:01.122839 8824 ts_tablet_manager.cc:1918] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:01.122913 8824 log.cc:1196] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/743aeb5d446947c3a5a3d0c9fa3b74c3
I20260925 10:13:01.123175 8824 ts_tablet_manager.cc:1939] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P c7452445fc434f149ec3850072775f33: Deleting consensus metadata
I20260925 10:13:01.123414 8826 ts_tablet_manager.cc:1918] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:01.123484 8826 log.cc:1196] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/743aeb5d446947c3a5a3d0c9fa3b74c3
I20260925 10:13:01.123648 8319 catalog_manager.cc:4841] TS c7452445fc434f149ec3850072775f33 (127.3.152.195:40775): tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 (table test-table [id=8713beff6e1f41fea8b10d4b6eb1f08e]) successfully deleted
I20260925 10:13:01.123742 8826 ts_tablet_manager.cc:1939] T 743aeb5d446947c3a5a3d0c9fa3b74c3 P 24e4a7268398488292ec89004acf1c3f: Deleting consensus metadata
I20260925 10:13:01.124326 8319 catalog_manager.cc:4841] TS 24e4a7268398488292ec89004acf1c3f (127.3.152.194:44891): tablet 743aeb5d446947c3a5a3d0c9fa3b74c3 (table test-table [id=8713beff6e1f41fea8b10d4b6eb1f08e]) successfully deleted
W20260925 10:13:01.124886 8318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:36559 (error 108)
I20260925 10:13:01.125188 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8527
I20260925 10:13:01.125430 8799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.125310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:36559 (local address 127.0.0.1:54088)
0925 10:13:01.125355 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.125394 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:13:01.125552 8318 catalog_manager.cc:4584] TS e58cea93d60244a18a3dde5cfedc48f9 (127.3.152.193:36559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 743aeb5d446947c3a5a3d0c9fa3b74c3: Network error: Client connection negotiation failed: client connection to 127.3.152.193:36559: connect: Connection refused (error 111)
I20260925 10:13:01.130447 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8659
I20260925 10:13:01.134929 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8302
2026-09-25T10:13:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (612 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 10:13:01.155421 3683 mini_oidc.cc:152] Starting JWKS server
I20260925 10:13:01.156641 3683 webserver.cc:460] Webserver started at https://127.0.0.1:37785/ using document root <none> and password file <none>
I20260925 10:13:01.156773 3683 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:13:01.156965 3683 webserver.cc:460] Webserver started at http://127.0.0.1:35517/ using document root <none> and password file <none>
2026-09-25T10:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:01Z Disabled control of system clock
I20260925 10:13:01.165047 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35447
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:35447
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37785/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/testchainca.cert with env {}
W20260925 10:13:01.247828 8833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:01.248118 8833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:01.248144 8833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:01.249424 8833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:01.249470 8833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:01.249483 8833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:01.249496 8833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:01.249508 8833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:13:01.249521 8833 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:13:01.249532 8833 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37785/jwks.json
I20260925 10:13:01.251092 8833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:35447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37785/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:01.251298 8833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:01.251461 8833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:01.256704 8838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.256685 8839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.256685 8841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.256742 8833 server_base.cc:1034] running on GCE node
I20260925 10:13:01.257192 8833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:01.257539 8833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:01.258724 8833 hybrid_clock.cc:648] HybridClock initialized: now 1790331181258710 us; error 44 us; skew 500 ppm
I20260925 10:13:01.260228 8833 webserver.cc:460] Webserver started at http://127.3.152.254:44283/ using document root <none> and password file <none>
I20260925 10:13:01.260483 8833 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:01.260551 8833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:01.260697 8833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:01.261822 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "74c45d8fcf784242882fa96ccef88d15"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "b4c7e1fdcc5baf892cb5f7a13bd3fa8f"
server_key_iv: "3641410095c97eae25006b5e66893f28"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.262197 8833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "74c45d8fcf784242882fa96ccef88d15"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "b4c7e1fdcc5baf892cb5f7a13bd3fa8f"
server_key_iv: "3641410095c97eae25006b5e66893f28"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.263551 8833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:01.264396 8847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.264601 8833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:13:01.264703 8833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "74c45d8fcf784242882fa96ccef88d15"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "b4c7e1fdcc5baf892cb5f7a13bd3fa8f"
server_key_iv: "3641410095c97eae25006b5e66893f28"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.264796 8833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:01.290277 8833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:01.290635 8833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:01.290819 8833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:01.294946 8833 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:35447
I20260925 10:13:01.295024 8899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:35447 every 8 connection(s)
I20260925 10:13:01.295382 8833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:01.296468 8900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.299268 8900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: Bootstrap starting.
I20260925 10:13:01.299716 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8833
I20260925 10:13:01.299835 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:01.300063 3683 external_mini_cluster.cc:1428] Setting key 9eedcbd7e67185a3069fdd8b11f9d0a5
I20260925 10:13:01.300122 8900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.300498 8900 log.cc:824] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.301331 8900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: No bootstrap required, opened a new log
I20260925 10:13:01.303864 8900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } }
I20260925 10:13:01.304095 8900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.304124 8900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c45d8fcf784242882fa96ccef88d15, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.304282 8900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } }
I20260925 10:13:01.304355 8900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:01.304396 8900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:01.304446 8900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.305199 8900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } }
I20260925 10:13:01.305311 8900 leader_election.cc:302] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c45d8fcf784242882fa96ccef88d15; no voters:
I20260925 10:13:01.305497 8900 leader_election.cc:288] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:01.305616 8905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:01.305811 8905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [term 1 LEADER]: Becoming Leader. State: Replica: 74c45d8fcf784242882fa96ccef88d15, State: Running, Role: LEADER
I20260925 10:13:01.305860 8900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:01.306005 8905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } }
I20260925 10:13:01.306385 8905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c45d8fcf784242882fa96ccef88d15. Latest consensus state: current_term: 1 leader_uuid: "74c45d8fcf784242882fa96ccef88d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } } }
I20260925 10:13:01.306475 8905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.307104 8907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c45d8fcf784242882fa96ccef88d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c45d8fcf784242882fa96ccef88d15" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35447 } } }
I20260925 10:13:01.307207 8907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.307947 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.300588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.3.152.254:35447)
0925 10:13:01.300809 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.300819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.300850 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:13:01.301513 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.301523 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.301603 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.301767 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.302005 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.303458 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.304786 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.305477 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.305619 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.306538 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.306542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.306542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.306889 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.307127 (+ 238us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.307182 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.307270 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.307605 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.307694 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
W20260925 10:13:01.308568 8920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:13:01.308627 8920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:13:01.308678 8921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:01.308866 8921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:01.309012 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:13:01.310418 8921 catalog_manager.cc:1347] Generated new cluster ID: db2d9de2ea0b4b928e5467efa9f783e6
I20260925 10:13:01.310487 8921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:01.314167 8921 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:01.314646 8921 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:01.323912 8921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c45d8fcf784242882fa96ccef88d15: Generated new TSK 0
I20260925 10:13:01.324175 8921 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:13:01.359400 8930 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 10:13:01.359534 8930 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:13:01.359827 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.336966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.3.152.254:35447)
0925 10:13:01.337046 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.337048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.337233 (+ 185us) server_negotiation.cc:407] Connection header received
0925 10:13:01.337392 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.337394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.337406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.337487 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:13:01.337728 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.338240 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.339094 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.339306 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.339385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.339622 (+ 237us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:13:01.359558 (+ 19936us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:13:01.359747 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 10:13:01.359920 8903 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 10:13:01.360293 8929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:01.336896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35447 (local address 127.0.0.1:44984)
0925 10:13:01.337142 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.337173 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.337257 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.337516 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.337517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.337520 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:13:01.337602 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.337604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.338334 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.338336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.338985 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.338988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.339388 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.339389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.339455 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.339485 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:13:01.359787 (+ 20302us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 10:13:01.359916 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.152.254:35447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.152.254:35447: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37785/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 10:13:01.360849 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8833
2026-09-25T10:13:01Z chronyd exiting
I20260925 10:13:01.397140 3683 test_util.cc:175] -----------------------------------------------
I20260925 10:13:01.397257 3683 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790331159020724-3683-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (242 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 10:13:01.398072 3683 mini_oidc.cc:152] Starting JWKS server
I20260925 10:13:01.398800 3683 webserver.cc:460] Webserver started at https://127.0.0.1:46735/ using document root <none> and password file <none>
I20260925 10:13:01.398871 3683 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:13:01.399055 3683 webserver.cc:460] Webserver started at http://127.0.0.1:40823/ using document root <none> and password file <none>
2026-09-25T10:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:01Z Disabled control of system clock
I20260925 10:13:01.406983 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46341
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:46341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46735/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/testchainca.cert with env {}
W20260925 10:13:01.489076 8936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:01.489348 8936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:01.489387 8936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:01.490619 8936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:01.490695 8936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:01.490717 8936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:01.490739 8936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:01.490760 8936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:13:01.490790 8936 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:13:01.490813 8936 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46735/jwks.json
I20260925 10:13:01.492463 8936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:46341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46735/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:01.492748 8936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:01.492972 8936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:01.498476 8942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.498558 8944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.498590 8936 server_base.cc:1034] running on GCE node
W20260925 10:13:01.498555 8941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.498942 8936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:01.499245 8936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:01.500412 8936 hybrid_clock.cc:648] HybridClock initialized: now 1790331181500383 us; error 43 us; skew 500 ppm
I20260925 10:13:01.501649 8936 webserver.cc:460] Webserver started at http://127.3.152.254:36089/ using document root <none> and password file <none>
I20260925 10:13:01.501910 8936 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:01.501981 8936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:01.502130 8936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:01.503206 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "b65af21844fa4648bd7bd23bc7026c20"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1db00388be9f0075c18ef8e9d1162f18"
server_key_iv: "020be7317ab4c54536cdf942fd00f651"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.503576 8936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "b65af21844fa4648bd7bd23bc7026c20"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1db00388be9f0075c18ef8e9d1162f18"
server_key_iv: "020be7317ab4c54536cdf942fd00f651"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.505112 8936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:13:01.505913 8950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.506099 8936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:01.506206 8936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "b65af21844fa4648bd7bd23bc7026c20"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1db00388be9f0075c18ef8e9d1162f18"
server_key_iv: "020be7317ab4c54536cdf942fd00f651"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.506286 8936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:01.520604 8936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:01.520970 8936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:01.521133 8936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:01.525348 8936 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:46341
I20260925 10:13:01.525406 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:46341 every 8 connection(s)
I20260925 10:13:01.525795 8936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:01.526816 9003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.529349 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20: Bootstrap starting.
I20260925 10:13:01.529903 9003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.530156 9003 log.cc:824] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.530865 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20: No bootstrap required, opened a new log
I20260925 10:13:01.531635 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 8936
I20260925 10:13:01.531769 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:01.531941 3683 external_mini_cluster.cc:1428] Setting key 379a29a294b52a5feba4d2c3fb3c0532
I20260925 10:13:01.532888 9003 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } }
I20260925 10:13:01.533064 9003 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.533106 9003 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65af21844fa4648bd7bd23bc7026c20, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.533253 9003 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } }
I20260925 10:13:01.533353 9003 raft_consensus.cc:388] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:01.533380 9003 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:01.533450 9003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.534032 9003 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } }
I20260925 10:13:01.534119 9003 leader_election.cc:302] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b65af21844fa4648bd7bd23bc7026c20; no voters:
I20260925 10:13:01.534297 9003 leader_election.cc:288] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:01.534420 9008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:01.534606 9008 raft_consensus.cc:686] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [term 1 LEADER]: Becoming Leader. State: Replica: b65af21844fa4648bd7bd23bc7026c20, State: Running, Role: LEADER
I20260925 10:13:01.534639 9003 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:01.534693 9008 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } }
I20260925 10:13:01.535029 9008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65af21844fa4648bd7bd23bc7026c20. Latest consensus state: current_term: 1 leader_uuid: "b65af21844fa4648bd7bd23bc7026c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } } }
I20260925 10:13:01.535068 9009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b65af21844fa4648bd7bd23bc7026c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65af21844fa4648bd7bd23bc7026c20" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46341 } } }
I20260925 10:13:01.535108 9008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.535115 9009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.536000 9015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:01.536566 9015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:01.538288 9015 catalog_manager.cc:1347] Generated new cluster ID: 50951b8cabdb45b789fa09baf6d59b96
I20260925 10:13:01.538340 9015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:01.538357 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.532510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.3.152.254:46341)
0925 10:13:01.532716 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.532724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.532749 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:13:01.532874 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.532879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.532934 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.533047 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.533333 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.534923 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.536176 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.536703 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.536826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.537596 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.537600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.537600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.537807 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.537851 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.537866 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.537929 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.538054 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.538135 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 10:13:01.539458 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:13:01.550503 9015 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:01.551337 9015 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:01.556954 9015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b65af21844fa4648bd7bd23bc7026c20: Generated new TSK 0
I20260925 10:13:01.557183 9015 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 10:13:01.592344 9033 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 10:13:01.592440 9033 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 10:13:01.592787 9007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.570697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.3.152.254:46341)
0925 10:13:01.570785 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.570787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.570796 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:01.570846 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.570847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.570857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.570943 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:13:01.571085 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.571594 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.572455 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.572650 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.572724 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.572928 (+ 204us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:13:01.592523 (+ 19595us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:13:01.592696 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47958: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 10:13:01.592857 9007 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47958: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 10:13:01.592876 9032 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:01.570467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:46341 (local address 127.0.0.1:47958)
0925 10:13:01.570643 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.570656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.570762 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.570940 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.570941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.570943 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:13:01.571012 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.571014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.571692 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.571693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.572364 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.572366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.572737 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.572738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.572799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.572804 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:13:01.592640 (+ 19836us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 10:13:01.592759 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.152.254:46341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46735/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260925 10:13:01.593144 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 8936
2026-09-25T10:13:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 10:13:01.631276 3683 mini_oidc.cc:152] Starting JWKS server
I20260925 10:13:01.631937 3683 webserver.cc:460] Webserver started at https://127.0.0.1:40923/ using document root <none> and password file <none>
I20260925 10:13:01.632071 3683 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 10:13:01.632265 3683 webserver.cc:460] Webserver started at http://127.0.0.1:46753/ using document root <none> and password file <none>
2026-09-25T10:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:01Z Disabled control of system clock
I20260925 10:13:01.640347 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41517
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41517
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:13:01.722878 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:01.723100 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:01.723124 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:01.724370 9039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:01.724413 9039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:01.724427 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:01.724439 9039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:01.724452 9039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:13:01.724464 9039 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:13:01.724475 9039 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40923/jwks.json
I20260925 10:13:01.725909 9039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:01.726199 9039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:01.726521 9039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:01.731570 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.731630 9039 server_base.cc:1034] running on GCE node
W20260925 10:13:01.731715 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.731748 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.732118 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:01.732482 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:01.733644 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790331181733623 us; error 44 us; skew 500 ppm
I20260925 10:13:01.734885 9039 webserver.cc:460] Webserver started at http://127.3.152.254:46633/ using document root <none> and password file <none>
I20260925 10:13:01.735150 9039 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:01.735221 9039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:01.735391 9039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:01.736557 9039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "fb15eb9f312542ec85738bbf8979f190"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1d12227c2867179614cb48a21be210b1"
server_key_iv: "fe6a1c39c6075da8cd40267fc7926fa3"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.736933 9039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "fb15eb9f312542ec85738bbf8979f190"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1d12227c2867179614cb48a21be210b1"
server_key_iv: "fe6a1c39c6075da8cd40267fc7926fa3"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.738310 9039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:01.739022 9053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.739207 9039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:01.739337 9039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "fb15eb9f312542ec85738bbf8979f190"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "1d12227c2867179614cb48a21be210b1"
server_key_iv: "fe6a1c39c6075da8cd40267fc7926fa3"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.739424 9039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:01.754673 9039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:01.755024 9039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:01.755184 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:01.759322 9039 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41517
I20260925 10:13:01.759361 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41517 every 8 connection(s)
I20260925 10:13:01.759755 9039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:01.760494 9106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.763144 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190: Bootstrap starting.
I20260925 10:13:01.763743 9106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.764055 9106 log.cc:824] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.764750 9106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190: No bootstrap required, opened a new log
I20260925 10:13:01.765236 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9039
I20260925 10:13:01.765352 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:01.765554 3683 external_mini_cluster.cc:1428] Setting key 37380856024d3dbc3ee1628831c83a9b
I20260925 10:13:01.767328 9106 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } }
I20260925 10:13:01.767495 9106 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:01.767541 9106 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb15eb9f312542ec85738bbf8979f190, State: Initialized, Role: FOLLOWER
I20260925 10:13:01.767694 9106 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } }
I20260925 10:13:01.767778 9106 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:01.767822 9106 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:01.767868 9106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:01.768520 9106 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } }
I20260925 10:13:01.768644 9106 leader_election.cc:302] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb15eb9f312542ec85738bbf8979f190; no voters:
I20260925 10:13:01.768849 9106 leader_election.cc:288] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:01.768883 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:01.769053 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [term 1 LEADER]: Becoming Leader. State: Replica: fb15eb9f312542ec85738bbf8979f190, State: Running, Role: LEADER
I20260925 10:13:01.769119 9106 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:01.769151 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } }
I20260925 10:13:01.769483 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb15eb9f312542ec85738bbf8979f190. Latest consensus state: current_term: 1 leader_uuid: "fb15eb9f312542ec85738bbf8979f190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } } }
I20260925 10:13:01.769486 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb15eb9f312542ec85738bbf8979f190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb15eb9f312542ec85738bbf8979f190" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41517 } } }
I20260925 10:13:01.769599 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.769749 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:01.769871 9118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:01.770382 9118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:01.772022 9118 catalog_manager.cc:1347] Generated new cluster ID: 2f1cc4b4c5b64adfbe9d5749cdae71e4
I20260925 10:13:01.772078 9118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:01.772233 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.766100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.3.152.254:41517)
0925 10:13:01.766323 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.766332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.766357 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:13:01.766818 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.766825 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.766890 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.767009 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.767259 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.768642 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.769831 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.770526 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.770629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.771460 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.771465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.771466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.771684 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.771729 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.771747 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.771830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.771907 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.772035 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260925 10:13:01.773176 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:13:01.788442 9118 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:01.789186 9118 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:01.800801 9118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb15eb9f312542ec85738bbf8979f190: Generated new TSK 0
I20260925 10:13:01.801060 9118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:13:01.820663 9110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.816286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.3.152.254:41517)
0925 10:13:01.816352 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.816354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.816463 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:13:01.816517 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.816518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.816532 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.816627 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:13:01.816741 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.817559 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.818309 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.818627 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.818699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.818868 (+ 169us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:13:01.820410 (+ 1542us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:13:01.820548 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 10:13:01.820735 9110 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54218: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 10:13:01.822523 9135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:01.816180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41517 (local address 127.0.0.1:54218)
0925 10:13:01.816372 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:01.816383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:01.816451 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:01.816631 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:01.816632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:01.816634 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:13:01.816685 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.816687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.817638 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.817638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.818225 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:01.818226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.818709 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.818710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:01.818761 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.818765 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:13:01.820507 (+ 1742us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 10:13:01.822424 (+ 1917us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.152.254:41517: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40923/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1716,"thread_start_us":121,"threads_started":1}
I20260925 10:13:01.822856 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9039
E20260925 10:13:01.831667 9136 webserver.cc:533] Webserver: sslize failed:
E20260925 10:13:01.831753 9136 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T10:13:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T10:13:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:01Z Disabled control of system clock
I20260925 10:13:01.868290 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42749
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:42749
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/testchainca.cert with env {}
W20260925 10:13:01.951225 9140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:01.951460 9140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:01.951485 9140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:01.952869 9140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:01.952948 9140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:01.952961 9140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:01.952975 9140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:01.952991 9140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 10:13:01.953008 9140 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 10:13:01.953019 9140 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 10:13:01.954588 9140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:42749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:01.954808 9140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:01.955010 9140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:13:01.960402 9140 server_base.cc:1034] running on GCE node
W20260925 10:13:01.960358 9145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.960366 9146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:01.960377 9148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:01.960896 9140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:01.961243 9140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:01.962440 9140 hybrid_clock.cc:648] HybridClock initialized: now 1790331181962424 us; error 35 us; skew 500 ppm
I20260925 10:13:01.963886 9140 webserver.cc:460] Webserver started at http://127.3.152.254:37515/ using document root <none> and password file <none>
I20260925 10:13:01.964197 9140 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:01.964284 9140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:01.964447 9140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:01.965587 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "3abfd778caf14f81b39d05134d899907"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "bc6b7a2feac2563a2051155c70e75f6b"
server_key_iv: "6b87498702c996f7c679e01e9515f97d"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.965979 9140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "3abfd778caf14f81b39d05134d899907"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "bc6b7a2feac2563a2051155c70e75f6b"
server_key_iv: "6b87498702c996f7c679e01e9515f97d"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.967353 9140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:13:01.968376 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:01.968673 9140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:01.968798 9140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "3abfd778caf14f81b39d05134d899907"
format_stamp: "Formatted at 2026-09-25 10:13:01 on dist-test-slave-dzwq"
server_key: "bc6b7a2feac2563a2051155c70e75f6b"
server_key_iv: "6b87498702c996f7c679e01e9515f97d"
server_key_version: "encryptionkey@0"
I20260925 10:13:01.968883 9140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:01.986132 9140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:01.986637 9140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:01.986845 9140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:01.991449 9140 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:42749
I20260925 10:13:01.991500 9206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:42749 every 8 connection(s)
I20260925 10:13:01.991936 9140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:01.992952 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9140
I20260925 10:13:01.993088 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:01.993204 9207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:01.993351 3683 external_mini_cluster.cc:1428] Setting key 96415005c0e87c100a7b3f765acd7541
I20260925 10:13:01.996699 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907: Bootstrap starting.
I20260925 10:13:01.997515 9207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:01.997862 9207 log.cc:824] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:01.998731 9207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907: No bootstrap required, opened a new log
I20260925 10:13:02.000006 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:01.994028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.3.152.254:42749)
0925 10:13:01.994288 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:01.994297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:01.994331 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:13:01.994850 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:01.994858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:01.994931 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:01.995077 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:01.995399 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.996545 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.997727 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:01.998306 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:01.998464 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:01.999200 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:01.999205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:01.999206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:01.999421 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:01.999467 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:01.999483 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:01.999555 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:01.999618 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:13:01.999720 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":119,"threads_started":1}
I20260925 10:13:02.001174 9207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } }
I20260925 10:13:02.001340 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:13:02.001377 9207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:02.001413 9207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3abfd778caf14f81b39d05134d899907, State: Initialized, Role: FOLLOWER
I20260925 10:13:02.001530 9207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } }
I20260925 10:13:02.001600 9207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:02.001627 9207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:02.001664 9207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:02.002378 9207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } }
I20260925 10:13:02.002516 9207 leader_election.cc:302] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3abfd778caf14f81b39d05134d899907; no voters:
I20260925 10:13:02.002712 9207 leader_election.cc:288] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:02.002758 9212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:02.002950 9212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [term 1 LEADER]: Becoming Leader. State: Replica: 3abfd778caf14f81b39d05134d899907, State: Running, Role: LEADER
I20260925 10:13:02.003024 9207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:02.003037 9212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } }
I20260925 10:13:02.003477 9214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3abfd778caf14f81b39d05134d899907. Latest consensus state: current_term: 1 leader_uuid: "3abfd778caf14f81b39d05134d899907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } } }
I20260925 10:13:02.003509 9213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3abfd778caf14f81b39d05134d899907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3abfd778caf14f81b39d05134d899907" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42749 } } }
I20260925 10:13:02.003618 9214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:02.003646 9213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:02.004133 9221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:02.004673 9221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:02.006664 9221 catalog_manager.cc:1347] Generated new cluster ID: 22d1540ef1864c97825f696da0f40317
I20260925 10:13:02.006726 9221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:02.025535 9221 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:02.026304 9221 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:02.035494 9221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3abfd778caf14f81b39d05134d899907: Generated new TSK 0
I20260925 10:13:02.035770 9221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:13:02.108996 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:02.048261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.3.152.254:42749)
0925 10:13:02.048320 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:02.048322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:02.048446 (+ 124us) server_negotiation.cc:407] Connection header received
0925 10:13:02.048535 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:02.048536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:02.048549 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:02.048619 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0925 10:13:02.048801 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:02.049317 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:02.050108 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:02.050331 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:02.050415 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:02.050620 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 10:13:02.108656 (+ 58036us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:13:02.108904 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48886: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 10:13:02.109091 9210 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48886: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 10:13:02.109092 9236 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:02.048198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42749 (local address 127.0.0.1:48886)
0925 10:13:02.048377 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:02.048387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:02.048465 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:02.048648 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:02.048649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:02.048651 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 10:13:02.048720 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:02.048722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:02.049400 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:02.049401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:02.050015 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:02.050017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:02.050426 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:02.050427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:02.050491 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:02.050500 (+ 9us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 10:13:02.108830 (+ 58330us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 10:13:02.108963 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.152.254:42749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260925 10:13:02.109534 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9140
2026-09-25T10:13:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (267 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:13:02 dist-test-slave-dzwq krb5kdc[9248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:13:02 dist-test-slave-dzwq krb5kdc[9248](info): set up 2 sockets
Sep 25 10:13:02 dist-test-slave-dzwq krb5kdc[9248](info): commencing operation
krb5kdc: starting...
W20260925 10:13:03.408986 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:13:03 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:13:03.534754 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43459
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:43459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:03.621265 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:03.621543 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:03.621585 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:03.622797 9261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:03.622877 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:03.622901 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:03.622923 9261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:03.622952 9261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:03.624517 9261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:43459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:03.624817 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:03.625052 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:03.630012 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:03.629987 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:03.629974 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:03.630019 9261 server_base.cc:1034] running on GCE node
I20260925 10:13:03.630411 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:03.630703 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:03.631873 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790331183631854 us; error 38 us; skew 500 ppm
Sep 25 10:13:03 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331183, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:03.634414 9261 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:13:03.634900 9261 webserver.cc:460] Webserver started at http://127.3.152.254:34681/ using document root <none> and password file <none>
I20260925 10:13:03.635143 9261 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:03.635212 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:03.635365 9261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:03.636493 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "8717086b26f04676b17533fa9d085472"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "e23436594842e62e994643d07231495e"
server_key_iv: "fc102f9f274aa435dc167e4f7a1c9703"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.636860 9261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "8717086b26f04676b17533fa9d085472"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "e23436594842e62e994643d07231495e"
server_key_iv: "fc102f9f274aa435dc167e4f7a1c9703"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.638245 9261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:03.639036 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:03.639338 9261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:03.639441 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8717086b26f04676b17533fa9d085472"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "e23436594842e62e994643d07231495e"
server_key_iv: "fc102f9f274aa435dc167e4f7a1c9703"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.639528 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:03.658980 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:03.661166 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:03.661351 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:03.665627 9261 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:43459
I20260925 10:13:03.665676 9329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:43459 every 8 connection(s)
I20260925 10:13:03.666059 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:03.666985 9330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:03.669797 9330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472: Bootstrap starting.
I20260925 10:13:03.670235 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9261
I20260925 10:13:03.670360 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:03.670390 9330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:03.670555 3683 external_mini_cluster.cc:1428] Setting key c81e1c736268cc04b36c69fa581b6374
I20260925 10:13:03.670742 9330 log.cc:824] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:03.671556 9330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472: No bootstrap required, opened a new log
I20260925 10:13:03.674075 9330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } }
I20260925 10:13:03.674259 9330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:03.674306 9330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8717086b26f04676b17533fa9d085472, State: Initialized, Role: FOLLOWER
I20260925 10:13:03.674448 9330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } }
I20260925 10:13:03.674549 9330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:03.674589 9330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:03.674643 9330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:03.675277 9330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } }
I20260925 10:13:03.675385 9330 leader_election.cc:302] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8717086b26f04676b17533fa9d085472; no voters:
I20260925 10:13:03.675551 9330 leader_election.cc:288] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:03.675599 9335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:03.675771 9335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [term 1 LEADER]: Becoming Leader. State: Replica: 8717086b26f04676b17533fa9d085472, State: Running, Role: LEADER
I20260925 10:13:03.675817 9330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:03.675858 9335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } }
I20260925 10:13:03.676201 9336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8717086b26f04676b17533fa9d085472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } } }
I20260925 10:13:03.676285 9336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:03.676501 9337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8717086b26f04676b17533fa9d085472. Latest consensus state: current_term: 1 leader_uuid: "8717086b26f04676b17533fa9d085472" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8717086b26f04676b17533fa9d085472" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43459 } } }
I20260925 10:13:03.676810 9337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:03.676966 9340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:03.677544 9340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:13:03 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:03.679562 9340 catalog_manager.cc:1347] Generated new cluster ID: a570b21616a643fca9eaeec24a6d90b3
I20260925 10:13:03.679616 9340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:03.682492 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:03.671075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.3.152.254:43459)
0925 10:13:03.671225 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:03.671235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:03.671269 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:13:03.671875 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:03.671884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:03.671951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:03.672133 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:03.672739 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.674226 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.675103 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.675760 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.677523 (+ 1763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:03.678920 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:03.678924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:03.678924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:03.678935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:03.680895 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:03.681290 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:03.681293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:03.681294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:03.681391 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:03.681604 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:03.681605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:03.681605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:03.681854 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:03.682013 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:03.682130 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 10:13:03.682271 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260925 10:13:03.732347 9340 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:03.733147 9340 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:03.740469 9340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8717086b26f04676b17533fa9d085472: Generated new TSK 0
I20260925 10:13:03.740692 9340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:13:03.771075 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:03.853811 9356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:03.854028 9356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:03.854053 9356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:03.855279 9356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:03.855329 9356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:03.855355 9356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:03.856974 9356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:03.857240 9356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:03.857508 9356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:03.862533 9362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:03.862705 9361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:03.862725 9364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:03.862927 9356 server_base.cc:1034] running on GCE node
I20260925 10:13:03.863114 9356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:03.863317 9356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:03.864454 9356 hybrid_clock.cc:648] HybridClock initialized: now 1790331183864439 us; error 28 us; skew 500 ppm
Sep 25 10:13:03 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331183, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:03.867195 9356 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:13:03.867741 9356 webserver.cc:460] Webserver started at http://127.3.152.193:44295/ using document root <none> and password file <none>
I20260925 10:13:03.868026 9356 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:03.868096 9356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:03.868201 9356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:03.869299 9356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "205708b950bb4a70b0ff5b3f3f0d717c"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "86b3ec5bde77266819ef6f553c56c1c8"
server_key_iv: "5b9ed9f2eaa9866b86d209d93ae0e380"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.869642 9356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "205708b950bb4a70b0ff5b3f3f0d717c"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "86b3ec5bde77266819ef6f553c56c1c8"
server_key_iv: "5b9ed9f2eaa9866b86d209d93ae0e380"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.871064 9356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:03.871941 9371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:03.872116 9356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:03.872233 9356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "205708b950bb4a70b0ff5b3f3f0d717c"
format_stamp: "Formatted at 2026-09-25 10:13:03 on dist-test-slave-dzwq"
server_key: "86b3ec5bde77266819ef6f553c56c1c8"
server_key_iv: "5b9ed9f2eaa9866b86d209d93ae0e380"
server_key_version: "encryptionkey@0"
I20260925 10:13:03.872313 9356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:03.890604 9356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:03.893133 9356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:03.893352 9356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:03.893674 9356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:03.894044 9356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:03.894104 9356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:03.894150 9356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:03.894197 9356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:03.900801 9356 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:33855
I20260925 10:13:03.900852 9484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:33855 every 8 connection(s)
I20260925 10:13:03.901223 9356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:03.906432 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9356
I20260925 10:13:03.906590 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:03.906843 3683 external_mini_cluster.cc:1428] Setting key ac99c671f45d0c4233c5457f167cebe2
Sep 25 10:13:03 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331183, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:03.911861 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:03.901994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:51911 (local address 127.3.152.254:43459)
0925 10:13:03.902077 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:03.902079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:03.902365 (+ 286us) server_negotiation.cc:407] Connection header received
0925 10:13:03.903322 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:03.903324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:03.903336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:03.903408 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:03.904166 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.904717 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.905754 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.905984 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.906049 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:03.908812 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:03.908813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:03.908814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:03.908822 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:03.910374 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:03.910828 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:03.910829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:03.910829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:03.910890 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:03.911162 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:03.911163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:03.911163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:03.911283 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:03.911361 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:03.911651 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 10:13:03.911743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:13:03.912292 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:03.901984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:43459 (local address 127.3.152.193:51911)
0925 10:13:03.902172 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:03.902216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:03.903187 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:03.903487 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:03.903491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:03.903773 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:03.904035 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:03.904041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.904810 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.904811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:03.905652 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:03.905654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:03.906069 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:03.906069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:03.906174 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:03.907018 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:03.907031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:03.908706 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:03.910468 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:03.910476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:03.910479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:03.910756 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:03.910956 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:03.910957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:03.910957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:03.911075 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:03.911352 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:03.911359 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:03.911511 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 10:13:03.911628 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:13:03.911795 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260925 10:13:03.912940 9485 heartbeater.cc:347] Connected to a master server at 127.3.152.254:43459
I20260925 10:13:03.913043 9485 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:03.913250 9485 heartbeater.cc:511] Master 127.3.152.254:43459 requested a full tablet report, sending...
I20260925 10:13:03.913915 9293 ts_manager.cc:194] Registered new tserver with Master: 205708b950bb4a70b0ff5b3f3f0d717c (127.3.152.193:33855)
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:13:03.914611 9293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:51911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:13:03.934470 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:04.016194 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:04.016414 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:04.016439 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:04.017560 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:04.017596 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:04.017621 9490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:04.019119 9490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:04.019309 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:04.019482 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:04.024699 9496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:04.024847 9490 server_base.cc:1034] running on GCE node
W20260925 10:13:04.024701 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:04.024798 9495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:04.025144 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:04.025339 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:04.026505 9490 hybrid_clock.cc:648] HybridClock initialized: now 1790331184026490 us; error 28 us; skew 500 ppm
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.029209 9490 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:13:04.029640 9490 webserver.cc:460] Webserver started at http://127.3.152.194:44825/ using document root <none> and password file <none>
I20260925 10:13:04.029908 9490 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:04.029980 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:04.030134 9490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:04.031256 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "f9b39074dbe448cb9b2fdd8fefd100e4"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "be844e73a843631f726eef101efcb5f2"
server_key_iv: "d696c9827b25852368a72ea2970055c8"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.031646 9490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "f9b39074dbe448cb9b2fdd8fefd100e4"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "be844e73a843631f726eef101efcb5f2"
server_key_iv: "d696c9827b25852368a72ea2970055c8"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.033066 9490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:13:04.033842 9505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.034027 9490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:04.034139 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "f9b39074dbe448cb9b2fdd8fefd100e4"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "be844e73a843631f726eef101efcb5f2"
server_key_iv: "d696c9827b25852368a72ea2970055c8"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.034224 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:04.052296 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:04.054808 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:04.054979 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:04.055235 9490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:04.055573 9490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:04.055636 9490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.055684 9490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:04.055717 9490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.061561 9490 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:38593
I20260925 10:13:04.061607 9618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:38593 every 8 connection(s)
I20260925 10:13:04.061941 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:04.069797 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9490
I20260925 10:13:04.069901 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:04.070135 3683 external_mini_cluster.cc:1428] Setting key 94ae6459826949355844c53a34d69fd8
I20260925 10:13:04.072003 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.062711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:47925 (local address 127.3.152.254:43459)
0925 10:13:04.062785 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.062787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.062982 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:13:04.063914 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.063915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.063924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.064042 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.064729 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.065212 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.066202 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.066389 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.066445 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.068917 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:04.068918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:04.068919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:04.068926 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:04.070542 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.070988 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.070989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.070989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.071043 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.071332 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.071333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.071333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.071432 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:04.071482 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:04.071828 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 10:13:04.071898 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:13:04.072321 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.062694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:43459 (local address 127.3.152.194:47925)
0925 10:13:04.062845 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:04.062889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:04.063818 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:04.064110 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:04.064114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:04.064369 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:04.064564 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:04.064568 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.065312 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.065313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:04.066123 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:04.066125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.066491 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.066492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:04.066619 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.067214 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:04.067222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:04.068800 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:04.070606 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:04.070614 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:04.070616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:04.070894 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:04.071106 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:04.071107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:04.071108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:04.071266 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:04.071498 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:04.071504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:04.071692 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 10:13:04.071809 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:13:04.072025 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260925 10:13:04.072968 9619 heartbeater.cc:347] Connected to a master server at 127.3.152.254:43459
I20260925 10:13:04.073091 9619 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:04.073333 9619 heartbeater.cc:511] Master 127.3.152.254:43459 requested a full tablet report, sending...
I20260925 10:13:04.073774 9293 ts_manager.cc:194] Registered new tserver with Master: f9b39074dbe448cb9b2fdd8fefd100e4 (127.3.152.194:38593)
I20260925 10:13:04.074172 9293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:47925
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:13:04.097361 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:04.181310 9624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:04.181537 9624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:04.181564 9624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:04.182806 9624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:04.182852 9624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:04.182879 9624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:04.184443 9624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:43459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:04.184662 9624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:04.184850 9624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:04.190074 9632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:04.190189 9624 server_base.cc:1034] running on GCE node
W20260925 10:13:04.190065 9630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:04.190080 9629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:04.190456 9624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:04.190714 9624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:04.191898 9624 hybrid_clock.cc:648] HybridClock initialized: now 1790331184191888 us; error 41 us; skew 500 ppm
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.194473 9624 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:13:04.194938 9624 webserver.cc:460] Webserver started at http://127.3.152.195:45271/ using document root <none> and password file <none>
I20260925 10:13:04.195142 9624 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:04.195183 9624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:04.195277 9624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:04.196380 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "f52cfc60f2ec453cb4f05317a652a307"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "106bec46e7bd84d633759a77fd6cf6c3"
server_key_iv: "fdefcb344b55bf1a705c55192e161fba"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.196678 9624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "f52cfc60f2ec453cb4f05317a652a307"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "106bec46e7bd84d633759a77fd6cf6c3"
server_key_iv: "fdefcb344b55bf1a705c55192e161fba"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.197981 9624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:04.198740 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.198984 9624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:04.199059 9624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "f52cfc60f2ec453cb4f05317a652a307"
format_stamp: "Formatted at 2026-09-25 10:13:04 on dist-test-slave-dzwq"
server_key: "106bec46e7bd84d633759a77fd6cf6c3"
server_key_iv: "fdefcb344b55bf1a705c55192e161fba"
server_key_version: "encryptionkey@0"
I20260925 10:13:04.199142 9624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:04.206506 9624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:04.208772 9624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:04.208923 9624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:04.209157 9624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:04.209475 9624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:04.209532 9624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.209574 9624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:04.209601 9624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:04.215525 9624 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:36181
I20260925 10:13:04.215569 9752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:36181 every 8 connection(s)
I20260925 10:13:04.215907 9624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:04.222461 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9624
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:04.222651 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:04.222851 3683 external_mini_cluster.cc:1428] Setting key 3a41c66ccd97aefc195fb05dd746dce9
I20260925 10:13:04.226401 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.216803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:52235 (local address 127.3.152.254:43459)
0925 10:13:04.216876 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.216878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.217092 (+ 214us) server_negotiation.cc:407] Connection header received
0925 10:13:04.218068 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.218069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.218080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.218142 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.218806 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.219311 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.220485 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.220666 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.220720 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.223130 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:04.223132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:04.223132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:04.223142 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:04.224762 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.225285 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.225286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.225286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.225353 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.225660 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.225661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.225662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.225812 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:04.225891 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:04.226208 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 10:13:04.226299 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:13:04.226586 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.216772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:43459 (local address 127.3.152.195:52235)
0925 10:13:04.216954 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:04.216997 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:04.217970 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:04.218209 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:04.218213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:04.218488 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:04.218692 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:04.218697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.219414 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.219415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:04.220405 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:04.220408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.220747 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.220748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:04.220857 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.221408 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:04.221417 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:04.223022 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:04.224943 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:04.224950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:04.224953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:04.225202 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:04.225434 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:04.225434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:04.225435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:04.225590 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:04.225892 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:04.225899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:04.226044 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:13:04.226149 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:13:04.226320 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 10:13:04.227079 9753 heartbeater.cc:347] Connected to a master server at 127.3.152.254:43459
I20260925 10:13:04.227207 9753 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:04.227399 9753 heartbeater.cc:511] Master 127.3.152.254:43459 requested a full tablet report, sending...
I20260925 10:13:04.227815 9293 ts_manager.cc:194] Registered new tserver with Master: f52cfc60f2ec453cb4f05317a652a307 (127.3.152.195:36181)
I20260925 10:13:04.228248 9293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:52235
I20260925 10:13:04.234781 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/test-admin.keytab.
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.254083 3683 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 10:13:04.255861 9760 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:04.255013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.3.152.193:33855)
0925 10:13:04.255184 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.255190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.255204 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:13:04.255241 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.255243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.255257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.255333 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.255755 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43924: BlockingRecv error: recv got EOF from 127.0.0.1:43924 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
W20260925 10:13:04.255875 3683 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.256982 3683 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.257931 3683 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:13:04.266036 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.258468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.3.152.193:33855)
0925 10:13:04.258526 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.258528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.258552 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:13:04.258620 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.258620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.258632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.258692 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.259098 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.259825 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.260605 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.260987 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.261053 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.263173 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:04.263181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:04.263181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:04.263189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:04.264764 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.265085 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.265088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.265088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.265155 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.265351 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.265352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.265352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.265511 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:04.265722 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:04.265784 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 10:13:04.265854 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 10:13:04.268142 9760 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:04.267524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.3.152.193:33855)
0925 10:13:04.267585 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.267587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.267665 (+ 78us) server_negotiation.cc:407] Connection header received
0925 10:13:04.267693 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.267693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.267705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.267770 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.268068 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43948: BlockingRecv error: recv got EOF from 127.0.0.1:43948 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 10:13:04.268314 3683 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc)
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.269353 3683 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.270251 3683 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:13:04.278662 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.270815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.3.152.193:33855)
0925 10:13:04.270879 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.270881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.270920 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:13:04.270966 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.270966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.270974 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.271026 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.271395 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.272028 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.272771 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.272970 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.273025 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.275511 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:04.275513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:04.275513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:04.275522 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:04.277370 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.277828 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.277829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.277829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.277889 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.278097 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.278098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.278098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.278242 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:04.278349 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:04.278469 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 10:13:04.278556 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 10:13:04.280455 9760 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:04.279793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.3.152.193:33855)
0925 10:13:04.279829 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.279831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.279916 (+ 85us) server_negotiation.cc:407] Connection header received
0925 10:13:04.280004 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.280005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.280015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.280077 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.280383 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43978: BlockingRecv error: recv got EOF from 127.0.0.1:43978 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 10:13:04.280578 3683 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790331159020724-3683-0/krb5kdc/krb5cc' of type FILE
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.281740 3683 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:04.282794 3683 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:13:04.291200 9760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:04.283394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.3.152.193:33855)
0925 10:13:04.283456 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:04.283458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:04.283559 (+ 101us) server_negotiation.cc:407] Connection header received
0925 10:13:04.283715 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:04.283716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:04.283726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:04.283778 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:04.284288 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.284787 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.285512 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:04.285742 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:04.285822 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:04.288393 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:04.288395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:04.288396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:04.288413 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:04.289923 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.290327 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.290329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.290329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.290394 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:04.290631 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:04.290632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:04.290632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:04.290751 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:04.290867 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:04.290983 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:13:04.291087 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 10:13:04.291952 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9356
I20260925 10:13:04.297802 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9490
I20260925 10:13:04.302414 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9624
I20260925 10:13:04.307058 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9261
2026-09-25T10:13:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2015 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9773](info): set up 2 sockets
Sep 25 10:13:04 dist-test-slave-dzwq krb5kdc[9773](info): commencing operation
krb5kdc: starting...
W20260925 10:13:05.380889 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:13:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab.
Entry for principal oryx/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab.
I20260925 10:13:05.504055 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33181
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab
--principal=oryx/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:05.590251 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:05.590481 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:05.590505 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:05.591639 9786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:05.591675 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:05.591689 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:05.591702 9786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:05.591714 9786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:05.593250 9786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab
--principal=oryx/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:05.593462 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:05.593652 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:05.598690 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:05.598788 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:05.599151 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:05.598987 9786 server_base.cc:1034] running on GCE node
I20260925 10:13:05.599627 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:05.599999 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:05.601183 9786 hybrid_clock.cc:648] HybridClock initialized: now 1790331185601173 us; error 28 us; skew 500 ppm
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:05.607470 9786 init.cc:377] Logged in from keytab as oryx/127.3.152.254@KRBTEST.COM (short username oryx)
I20260925 10:13:05.608089 9786 webserver.cc:460] Webserver started at http://127.3.152.254:44061/ using document root <none> and password file <none>
I20260925 10:13:05.608345 9786 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:05.608414 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:05.608554 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:05.609637 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "2a1134c071b14f8c8882da1d9e4e9328"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "8a8d9c5af8ffc80e8a1cb2cddd8e0fed"
server_key_iv: "45b83e4ec29f84e8f0a638325e2cda55"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.610044 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "2a1134c071b14f8c8882da1d9e4e9328"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "8a8d9c5af8ffc80e8a1cb2cddd8e0fed"
server_key_iv: "45b83e4ec29f84e8f0a638325e2cda55"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.611954 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:13:05.612761 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.612948 9786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:05.613049 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "2a1134c071b14f8c8882da1d9e4e9328"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "8a8d9c5af8ffc80e8a1cb2cddd8e0fed"
server_key_iv: "45b83e4ec29f84e8f0a638325e2cda55"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.613138 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:05.626230 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:05.628770 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:05.628937 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:05.633033 9786 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:33181
I20260925 10:13:05.633067 9853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:33181 every 8 connection(s)
I20260925 10:13:05.633431 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:05.634471 9854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:05.636945 9854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Bootstrap starting.
I20260925 10:13:05.637563 9854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:05.637872 9854 log.cc:824] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:05.638573 9854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: No bootstrap required, opened a new log
I20260925 10:13:05.640282 9854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } }
I20260925 10:13:05.640432 9854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:05.640456 9854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1134c071b14f8c8882da1d9e4e9328, State: Initialized, Role: FOLLOWER
I20260925 10:13:05.640635 9854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } }
I20260925 10:13:05.640648 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9786
I20260925 10:13:05.640725 9854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:05.640755 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:05.640779 9854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:05.640839 9854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:05.640950 3683 external_mini_cluster.cc:1428] Setting key a0a7b670d2d5e224a03698e7f7a425c7
I20260925 10:13:05.641947 9854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } }
I20260925 10:13:05.642091 9854 leader_election.cc:302] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a1134c071b14f8c8882da1d9e4e9328; no voters:
I20260925 10:13:05.642261 9854 leader_election.cc:288] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:05.642318 9859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:05.642446 9859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 LEADER]: Becoming Leader. State: Replica: 2a1134c071b14f8c8882da1d9e4e9328, State: Running, Role: LEADER
I20260925 10:13:05.642540 9854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:05.642541 9859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } }
I20260925 10:13:05.642906 9860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1134c071b14f8c8882da1d9e4e9328. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } }
I20260925 10:13:05.642997 9860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:05.642899 9861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } }
I20260925 10:13:05.643133 9861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:05.643271 9865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:05.643757 9865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:05.645069 9876 mvcc.cc:205] Tried to move back new op lower bound from 7333196536400678912 to 7333196536392409088. Current Snapshot: MvccSnapshot[applied={T|T < 7333196536400678912}]
I20260925 10:13:05.645430 9865 catalog_manager.cc:1347] Generated new cluster ID: 60234a734d98485e90f81b758c7f85cb
I20260925 10:13:05.645483 9865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:05.652545 9865 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:05.653230 9865 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:05.653266 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:05.641518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.3.152.254:33181)
0925 10:13:05.641708 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:05.641716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:05.641743 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:13:05.641836 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:05.641842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:05.641884 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:05.641994 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:05.642870 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.644556 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.645653 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.646239 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.646326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:05.649170 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:05.649174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:05.649174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:05.649187 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:05.651168 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:05.651546 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:05.651549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:05.651550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:05.651697 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:05.651894 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:05.651894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:05.651895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:05.652210 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:05.652360 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:05.652888 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 10:13:05.653039 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260925 10:13:05.656340 9865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Generated new TSK 0
I20260925 10:13:05.656538 9865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
I20260925 10:13:05.686936 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:05.770262 9880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:05.770469 9880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:05.770493 9880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:05.771638 9880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:05.771682 9880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:05.771708 9880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:05.773156 9880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:05.773355 9880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:05.773535 9880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:05.778631 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:05.778631 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:05.778738 9880 server_base.cc:1034] running on GCE node
W20260925 10:13:05.778659 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:05.779088 9880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:05.779343 9880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:05.780505 9880 hybrid_clock.cc:648] HybridClock initialized: now 1790331185780496 us; error 46 us; skew 500 ppm
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:05.783085 9880 init.cc:377] Logged in from keytab as oryx/127.3.152.193@KRBTEST.COM (short username oryx)
I20260925 10:13:05.783527 9880 webserver.cc:460] Webserver started at http://127.3.152.193:37739/ using document root <none> and password file <none>
I20260925 10:13:05.783763 9880 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:05.783830 9880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:05.784029 9880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:05.785089 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "8884c173f64d429c8e8d1976af0804bd"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "e0699ce126eda230d5adde0d3a51931a"
server_key_iv: "c190a1e7a8ebf79fee783494a9b869b6"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.785470 9880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "8884c173f64d429c8e8d1976af0804bd"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "e0699ce126eda230d5adde0d3a51931a"
server_key_iv: "c190a1e7a8ebf79fee783494a9b869b6"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.786872 9880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:05.787607 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.787791 9880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:05.787891 9880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "8884c173f64d429c8e8d1976af0804bd"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "e0699ce126eda230d5adde0d3a51931a"
server_key_iv: "c190a1e7a8ebf79fee783494a9b869b6"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.788000 9880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:05.807545 9880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:05.810081 9880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:05.810261 9880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:05.810518 9880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:05.810863 9880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:05.810920 9880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.810983 9880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:05.811024 9880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.817168 9880 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:37685
I20260925 10:13:05.817239 10008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:37685 every 8 connection(s)
I20260925 10:13:05.817626 9880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:05.822099 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 9880
I20260925 10:13:05.822216 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:05.822548 3683 external_mini_cluster.cc:1428] Setting key ca43b6cb0cc7881aff87f427107bb930
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:05.827905 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:05.818391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42321 (local address 127.3.152.254:33181)
0925 10:13:05.818463 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:05.818465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:05.818696 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:13:05.819180 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:05.819181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:05.819199 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:05.819259 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:05.820269 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.820832 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.821900 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.822273 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.822358 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:05.824930 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:05.824932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:05.824932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:05.824939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:05.826444 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:05.826885 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:05.826887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:05.826887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:05.826952 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:05.827209 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:05.827209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:05.827210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:05.827353 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:05.827432 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:05.827720 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:13:05.827812 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:13:05.828163 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:05.818393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:42321)
0925 10:13:05.818557 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:05.818602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:05.819090 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:05.819326 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:05.819329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:05.819916 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:05.820147 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:05.820152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.820935 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.820935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:05.821771 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:05.821772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.822368 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.822369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:05.822496 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:05.823103 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:05.823113 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:05.824816 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:05.826538 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:05.826547 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:05.826550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:05.826790 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:05.827028 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:05.827029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:05.827029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:05.827141 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:05.827424 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:05.827431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:05.827579 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:13:05.827687 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:13:05.827867 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260925 10:13:05.828701 10009 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:05.828830 10009 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:05.829013 10009 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:05.829572 9818 ts_manager.cc:194] Registered new tserver with Master: 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685)
I20260925 10:13:05.830192 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:42321
WARNING: no policy specified for oryx/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
I20260925 10:13:05.851143 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:05.934252 10014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:05.934463 10014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:05.934487 10014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:05.935701 10014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:05.935750 10014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:05.935777 10014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:05.937342 10014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:05.937601 10014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:05.937851 10014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:05.942909 10019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:05.942909 10022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:05.943085 10020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:05.943233 10014 server_base.cc:1034] running on GCE node
I20260925 10:13:05.943449 10014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:05.943697 10014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:05.944837 10014 hybrid_clock.cc:648] HybridClock initialized: now 1790331185944812 us; error 40 us; skew 500 ppm
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:05.947640 10014 init.cc:377] Logged in from keytab as oryx/127.3.152.194@KRBTEST.COM (short username oryx)
I20260925 10:13:05.948227 10014 webserver.cc:460] Webserver started at http://127.3.152.194:36381/ using document root <none> and password file <none>
I20260925 10:13:05.948483 10014 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:05.948555 10014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:05.948709 10014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:05.949803 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "3f50cb869fa94bc69e439a6a12821286"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "c03388c9e2f4d050c0acbafe1d91100f"
server_key_iv: "fca95c350b40e3dece272e7625dc59d9"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.950171 10014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "3f50cb869fa94bc69e439a6a12821286"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "c03388c9e2f4d050c0acbafe1d91100f"
server_key_iv: "fca95c350b40e3dece272e7625dc59d9"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.951691 10014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:05.952642 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.952823 10014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:13:05.952927 10014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "3f50cb869fa94bc69e439a6a12821286"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "c03388c9e2f4d050c0acbafe1d91100f"
server_key_iv: "fca95c350b40e3dece272e7625dc59d9"
server_key_version: "encryptionkey@0"
I20260925 10:13:05.953017 10014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:05.982035 10014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:05.984364 10014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:05.984531 10014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:05.984771 10014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:05.985088 10014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:05.985143 10014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.985190 10014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:05.985229 10014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:05.991339 10014 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:37495
I20260925 10:13:05.991384 10142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:37495 every 8 connection(s)
I20260925 10:13:05.991720 10014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:05.996481 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10014
I20260925 10:13:05.996644 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:05.996837 3683 external_mini_cluster.cc:1428] Setting key ea19a2e3c8defa7aea8690d437bb3a25
Sep 25 10:13:05 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:06.001729 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:05.992529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:43493 (local address 127.3.152.254:33181)
0925 10:13:05.992579 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:05.992581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:05.992812 (+ 231us) server_negotiation.cc:407] Connection header received
0925 10:13:05.993375 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:05.993377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:05.993389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:05.993445 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:05.994418 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.994937 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.996004 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.996222 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.996312 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:05.998752 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:05.998753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:05.998754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:05.998762 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.000305 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.000727 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.000728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.000728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.000791 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.001051 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.001052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.001053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.001174 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.001261 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.001538 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.001621 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:06.001938 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:05.992504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:43493)
0925 10:13:05.992689 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:05.992729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:05.993262 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:05.993531 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:05.993534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:05.994111 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:05.994318 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:05.994323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.995042 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.995052 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:05.995871 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:05.995873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:05.996324 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:05.996325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:05.996459 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:05.997053 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:05.997065 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:05.998642 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.000406 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.000413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.000415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.000645 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.000866 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.000867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.000867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.000983 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.001244 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.001250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.001404 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.001518 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.001684 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260925 10:13:06.002460 10143 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:06.002591 10143 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:06.002817 10143 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:06.003228 9818 ts_manager.cc:194] Registered new tserver with Master: 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
I20260925 10:13:06.003608 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:43493
WARNING: no policy specified for oryx/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
I20260925 10:13:06.023274 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:06.105862 10148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:06.106076 10148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:06.106102 10148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:06.107219 10148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:06.107257 10148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:06.107283 10148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:06.108824 10148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:06.109040 10148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:06.109229 10148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.114486 10153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.114423 10154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.114595 10148 server_base.cc:1034] running on GCE node
W20260925 10:13:06.114421 10156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.114881 10148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.115091 10148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.116254 10148 hybrid_clock.cc:648] HybridClock initialized: now 1790331186116244 us; error 46 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.119230 10148 init.cc:377] Logged in from keytab as oryx/127.3.152.195@KRBTEST.COM (short username oryx)
I20260925 10:13:06.119719 10148 webserver.cc:460] Webserver started at http://127.3.152.195:45583/ using document root <none> and password file <none>
I20260925 10:13:06.120045 10148 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.120118 10148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.120278 10148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:06.121435 10148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "65845fb86457409cbf24d3143cbe5494"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "7526d24b3a63bc35daa35054f536ef9d"
server_key_iv: "0c23dfa0bee959701aa2268dd25283d7"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.121811 10148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "65845fb86457409cbf24d3143cbe5494"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "7526d24b3a63bc35daa35054f536ef9d"
server_key_iv: "0c23dfa0bee959701aa2268dd25283d7"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.123239 10148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:06.124179 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.124372 10148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.124480 10148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "65845fb86457409cbf24d3143cbe5494"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "7526d24b3a63bc35daa35054f536ef9d"
server_key_iv: "0c23dfa0bee959701aa2268dd25283d7"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.124565 10148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.150930 10148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.153861 10148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:06.154036 10148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.154290 10148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:06.154650 10148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:06.154711 10148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.154755 10148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:06.154783 10148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.161062 10148 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:34029
I20260925 10:13:06.161113 10277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:34029 every 8 connection(s)
I20260925 10:13:06.161453 10148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:06.169041 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10148
I20260925 10:13:06.169168 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:06.169394 3683 external_mini_cluster.cc:1428] Setting key 5f0cf8611049961ff0897a7edf1cc5b7
I20260925 10:13:06.172529 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.162261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:60125 (local address 127.3.152.254:33181)
0925 10:13:06.162315 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.162317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.162653 (+ 336us) server_negotiation.cc:407] Connection header received
0925 10:13:06.163196 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.163198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.163210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.163266 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.164521 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.165032 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.166044 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.166241 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.166301 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.168780 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.168782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.168782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.168789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.170867 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.171340 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.171341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.171342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.171412 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.171719 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.171720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.171720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.171859 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.171952 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.172325 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.172428 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:13:06.172780 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.162220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:60125)
0925 10:13:06.162430 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.162474 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.163084 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.163381 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.163385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.164188 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.164404 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.164410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.165135 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.165136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.165955 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.165957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.166321 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.166322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.166423 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.166993 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.167001 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.168664 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.170970 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.170977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.170980 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.171251 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.171501 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.171502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.171502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.171634 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.171942 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.171949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.172152 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.172298 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.172487 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":134,"threads_started":1}
I20260925 10:13:06.173312 10278 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:06.173446 10278 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:06.173727 10278 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:06.174213 9818 ts_manager.cc:194] Registered new tserver with Master: 65845fb86457409cbf24d3143cbe5494 (127.3.152.195:34029)
I20260925 10:13:06.174698 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:60125
I20260925 10:13:06.181815 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.152.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.253.keytab.
Entry for principal oryx/127.3.152.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.253.keytab.
I20260925 10:13:06.213506 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:42821
--webserver_interface=127.3.152.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.253.keytab
--principal=oryx/127.3.152.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33181,127.3.152.253:42821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:06.298539 10283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:06.298815 10283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:06.298859 10283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:06.300220 10283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:06.300283 10283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:06.300307 10283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:06.300338 10283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:06.300364 10283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:06.301898 10283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33181,127.3.152.253:42821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.253.keytab
--principal=oryx/127.3.152.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:42821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.152.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
W20260925 10:13:06.302062 10283 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.152.254:33181,127.3.152.253:42821'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 10:13:06.302233 10283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:06.302471 10283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.307672 10288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.307679 10289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.307950 10283 server_base.cc:1034] running on GCE node
W20260925 10:13:06.307731 10291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.308306 10283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.308629 10283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.309808 10283 hybrid_clock.cc:648] HybridClock initialized: now 1790331186309797 us; error 33 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.312799 10283 init.cc:377] Logged in from keytab as oryx/127.3.152.253@KRBTEST.COM (short username oryx)
I20260925 10:13:06.313362 10283 webserver.cc:460] Webserver started at http://127.3.152.253:44255/ using document root <none> and password file <none>
I20260925 10:13:06.313603 10283 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.313653 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.313800 10283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:06.314906 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/instance:
uuid: "108023a6f92c4f74a2a889b0749a4de3"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "e77ea4752f28984166834677f32a0ec9"
server_key_iv: "4b4c4d442e891cee70233d096c72bb94"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.315338 10283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal/instance:
uuid: "108023a6f92c4f74a2a889b0749a4de3"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "e77ea4752f28984166834677f32a0ec9"
server_key_iv: "4b4c4d442e891cee70233d096c72bb94"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.316990 10283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:06.317895 10298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.318176 10283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:06.318284 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "108023a6f92c4f74a2a889b0749a4de3"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "e77ea4752f28984166834677f32a0ec9"
server_key_iv: "4b4c4d442e891cee70233d096c72bb94"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.318373 10283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.329823 10283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.332362 10283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:06.332482 10283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.336306 10283 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.253:42821
I20260925 10:13:06.336338 10350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.253:42821 every 8 connection(s)
I20260925 10:13:06.336827 10283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
I20260925 10:13:06.337867 10351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:06.338867 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10283
I20260925 10:13:06.340020 10351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.253@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
I20260925 10:13:06.350346 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.341067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.3.152.254:33181)
0925 10:13:06.341117 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.341119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.341252 (+ 133us) server_negotiation.cc:407] Connection header received
0925 10:13:06.341323 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.341324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.341337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.341396 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.342087 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.342556 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.343600 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.343846 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.343926 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.346817 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.346818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.346819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.346827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.348513 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.349149 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.349150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.349151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.349219 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.349484 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.349485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.349485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.349802 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.349855 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.350127 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.350227 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:06.350633 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.339572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.3.152.253:42821)
0925 10:13:06.339771 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.339781 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.339809 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:13:06.340331 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.340339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.340397 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.340521 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.341401 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.342572 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.343498 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.343981 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.344054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.346897 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.346903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.346903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.346909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.349098 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.349482 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.349485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.349485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.349539 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.349742 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.349743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.349743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.349999 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.350186 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.350273 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.350361 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
I20260925 10:13:06.350752 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.340985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46380)
0925 10:13:06.341138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.341178 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.341253 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.341418 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.341421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.341860 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.341997 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.342002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.342656 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.342657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.343511 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.343513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.343933 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.343934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.344053 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.344778 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.344786 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.346711 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.348598 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.348604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.348606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.349051 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.349302 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.349302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.349303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.349402 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.349867 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.349871 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.349991 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.350096 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.350277 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":21,"thread_start_us":104,"threads_started":1}
I20260925 10:13:06.351701 10351 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.253@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
I20260925 10:13:06.358349 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.352057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35756)
0925 10:13:06.352124 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.352136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.352200 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.352337 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.352338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.352472 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.352515 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.352516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.353156 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.353157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.353635 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.353636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.353994 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.353995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.354054 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.354450 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.354454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.355707 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.357321 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.357322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.357323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.357488 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.357692 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.357693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.357693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.357779 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.358023 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.358024 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.358061 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.358128 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.358225 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 10:13:06.358393 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.352077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.3.152.253:42821)
0925 10:13:06.352128 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.352130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.352184 (+ 54us) server_negotiation.cc:407] Connection header received
0925 10:13:06.352255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.352256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.352265 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.352334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.352595 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.353068 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.353724 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.353923 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.354001 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.355792 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.355794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.355794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.355802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.357152 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.357574 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.357575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.357575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.357629 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.357849 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.357850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.357850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.357960 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.358033 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.358155 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.358280 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":36}
I20260925 10:13:06.359776 10351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Bootstrap starting.
I20260925 10:13:06.360690 10351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:06.360986 10351 log.cc:824] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:06.361740 10351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: No bootstrap required, opened a new log
I20260925 10:13:06.363164 10351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } }
I20260925 10:13:06.363304 10351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:06.363328 10351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108023a6f92c4f74a2a889b0749a4de3, State: Initialized, Role: FOLLOWER
I20260925 10:13:06.363476 10351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } }
I20260925 10:13:06.363761 10357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } }
I20260925 10:13:06.363806 10351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:06.363893 10357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:13:06.365491 10368 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:13:06.365562 10368 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:13:06.367254 10283 master_runner.cc:428] Detected that this master 108023a6f92c4f74a2a889b0749a4de3 is joining an existing cluster
I20260925 10:13:06.367321 10283 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.152.253:42821
I20260925 10:13:06.375206 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.368571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35760)
0925 10:13:06.368685 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.368694 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.368747 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.368883 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.368883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.369171 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.369289 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.369291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.370183 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.370184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.370982 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.370983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.371324 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.371325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.371390 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.371890 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.371895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.372369 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.374150 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.374151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.374333 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.374550 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.374551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.374614 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.374863 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.374864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.374901 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.374969 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.375073 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260925 10:13:06.375211 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.368629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.3.152.253:42821)
0925 10:13:06.368670 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.368673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.368757 (+ 84us) server_negotiation.cc:407] Connection header received
0925 10:13:06.368791 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.368792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.368803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.368873 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.369372 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.370092 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.371034 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.371241 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.371315 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.372470 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.372472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.372472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.372479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.374039 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374418 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.374418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.374419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.374480 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374686 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.374686 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.374687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.374796 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.374870 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.374995 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.375128 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":21}
I20260925 10:13:06.375715 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.368332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46388)
0925 10:13:06.368482 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.368494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.368543 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.368842 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.368845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.369176 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.369289 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.369291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.369986 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.369988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.370895 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.370897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.371443 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.371444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.371512 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.372103 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.372108 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.372552 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.374425 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.374426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.374660 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.374874 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374874 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.374875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.374931 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.375212 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.375213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.375266 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.375330 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.375429 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260925 10:13:06.376088 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.368401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.3.152.254:33181)
0925 10:13:06.368693 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.368696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.368705 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:06.368731 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.368732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.368744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.368797 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.369394 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.369900 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.370991 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.371314 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.371386 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.372657 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.372658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.372659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.372666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.374333 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.374742 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.374743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.374744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.374802 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.375000 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.375001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.375001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.375130 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.375188 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.375888 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.376007 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.254@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
I20260925 10:13:06.387221 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.378349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.3.152.253:42821)
0925 10:13:06.378382 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.378384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.380210 (+ 1826us) server_negotiation.cc:407] Connection header received
0925 10:13:06.380316 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.380317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.380327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.380385 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.380826 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.381290 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.382091 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.382279 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.382352 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.384540 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.384541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.384541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.384549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.385976 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.386407 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.386408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.386409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.386467 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.386670 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.386671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.386671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.386782 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.386854 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.387045 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.387120 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:13:06.387264 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.378294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35766)
0925 10:13:06.380099 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.380145 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.380237 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.380391 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.380395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.380689 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.380741 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.380743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.381372 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.381373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.381993 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.381995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.382350 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.382351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.382448 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.382923 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.382927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.384449 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.386077 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.386085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.386088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.386314 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.386539 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.386540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.386540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.386605 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.386860 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.386869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.386921 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.387020 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.387148 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"thread_start_us":74,"threads_started":1}
I20260925 10:13:06.387825 9818 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.152.253:42821: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3"
member_type: NON_VOTER
last_known_addr {
host: "127.3.152.253"
port: 42821
}
attrs {
promote: true
}
}
dest_uuid: "2a1134c071b14f8c8882da1d9e4e9328"
cas_config_opid_index: -1
I20260925 10:13:06.388177 9818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } }
I20260925 10:13:06.388674 10378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.389082 10325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:06.389882 10378 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.152.253:42821
I20260925 10:13:06.390069 10325 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Refusing update from remote peer 2a1134c071b14f8c8882da1d9e4e9328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 10:13:06.390159 10283 master.cc:537] Master@127.3.152.253:42821 shutting down...
I20260925 10:13:06.390780 10378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:13:06.391690 9802 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 10:13:06.392629 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.392822 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:06.392835 10283 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:06.392926 10283 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:06.392983 10283 tablet_replica.cc:330] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: stopping tablet replica
W20260925 10:13:06.393254 9802 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 -> Peer 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253:42821): Couldn't send request to peer 108023a6f92c4f74a2a889b0749a4de3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:13:06.395522 10283 master.cc:552] Master@127.3.152.253:42821 shutdown complete.
I20260925 10:13:06.395570 10283 master_runner.cc:305] Clearing existing system tablet
I20260925 10:13:06.395699 10283 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.395747 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.396447 10283 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.396911 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.397027 10283 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.397080 10283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.397102 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "108023a6f92c4f74a2a889b0749a4de3"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "e77ea4752f28984166834677f32a0ec9"
server_key_iv: "4b4c4d442e891cee70233d096c72bb94"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.397725 10283 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:06.399194 10283 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:13:06.399266 10283 log.cc:1196] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 10:13:06.399477 10283 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Deleting consensus metadata
I20260925 10:13:06.399600 10283 master_runner.cc:315] Copying system tablet from 127.3.152.254:33181
I20260925 10:13:06.400280 10283 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.152.254:33181
I20260925 10:13:06.408322 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.400819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.3.152.254:33181)
0925 10:13:06.400868 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.400869 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.400919 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:13:06.401003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.401004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.401013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.401091 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.401515 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.402233 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.403326 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.403531 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.403607 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.405343 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.405345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.405346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.405352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.407049 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.407510 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.407511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.407512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.407577 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.407799 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.407800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.407800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.407915 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.408059 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.408166 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.408250 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:06.408380 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.400719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46396)
0925 10:13:06.400859 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.400870 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.400934 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.401130 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.401131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.401367 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.401430 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.401432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.402317 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.402318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.403207 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.403210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.403620 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.403621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.403697 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.404717 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.404722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.405189 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.407171 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.407173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.407173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.407413 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.407656 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.407657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.407657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.407727 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.408006 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.408007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.408058 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.408146 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.408260 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260925 10:13:06.409030 9838 tablet_copy_service.cc:140] P 2a1134c071b14f8c8882da1d9e4e9328: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 108023a6f92c4f74a2a889b0749a4de3 ({username='oryx', principal='oryx/127.3.152.253@KRBTEST.COM'} at 127.0.0.1:46396)
I20260925 10:13:06.409155 9838 tablet_copy_service.cc:161] P 2a1134c071b14f8c8882da1d9e4e9328: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 108023a6f92c4f74a2a889b0749a4de3 at {username='oryx', principal='oryx/127.3.152.253@KRBTEST.COM'} at 127.0.0.1:46396: session id = 108023a6f92c4f74a2a889b0749a4de3-00000000000000000000000000000000
I20260925 10:13:06.409999 9838 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:13:06.410646 10283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:13:06.411619 10283 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: tablet copy: Starting download of 0 data blocks...
I20260925 10:13:06.411844 10283 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: tablet copy: Starting download of 1 WAL segments...
I20260925 10:13:06.412807 10283 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:13:06.413647 9838 tablet_copy_service.cc:342] P 2a1134c071b14f8c8882da1d9e4e9328: Request end of tablet copy session 108023a6f92c4f74a2a889b0749a4de3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.152.253@KRBTEST.COM'} at 127.0.0.1:46396
I20260925 10:13:06.413712 9838 tablet_copy_service.cc:433] P 2a1134c071b14f8c8882da1d9e4e9328: ending tablet copy session 108023a6f92c4f74a2a889b0749a4de3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 108023a6f92c4f74a2a889b0749a4de3
I20260925 10:13:06.417028 10283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.418367 10392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.418318 10390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.418689 10283 server_base.cc:1034] running on GCE node
W20260925 10:13:06.418388 10389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.421053 10283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.421221 10283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.422351 10283 hybrid_clock.cc:648] HybridClock initialized: now 1790331186422333 us; error 39 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.423954 10283 init.cc:377] Logged in from keytab as oryx/127.3.152.253@KRBTEST.COM (short username oryx)
I20260925 10:13:06.424273 10283 webserver.cc:460] Webserver started at http://127.3.152.253:46655/ using document root <none> and password file <none>
I20260925 10:13:06.424399 10283 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.424461 10283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.425163 10283 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:06.425766 10399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.425933 10283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:06.426002 10283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "108023a6f92c4f74a2a889b0749a4de3"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "e77ea4752f28984166834677f32a0ec9"
server_key_iv: "4b4c4d442e891cee70233d096c72bb94"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.426066 10283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.434494 10283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.436421 10283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.439726 10283 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.253:42821
I20260925 10:13:06.440181 10283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
I20260925 10:13:06.440210 10451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.253:42821 every 8 connection(s)
I20260925 10:13:06.441148 10452 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:13:06.441685 10452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Bootstrap starting.
I20260925 10:13:06.445107 10452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:13:06.445401 10452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: Bootstrap complete.
I20260925 10:13:06.445654 10452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } }
I20260925 10:13:06.445758 10452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 108023a6f92c4f74a2a889b0749a4de3, State: Initialized, Role: LEARNER
I20260925 10:13:06.445801 10452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } }
I20260925 10:13:06.445997 10452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:06.446024 10455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.446172 10455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:13:06.447818 10466 catalog_manager.cc:1259] Loaded cluster ID: 60234a734d98485e90f81b758c7f85cb
I20260925 10:13:06.447916 10466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: loading cluster ID for follower catalog manager: success
I20260925 10:13:06.448863 10466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: acquiring CA information for follower catalog manager: success
I20260925 10:13:06.449170 10466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:13:06.459858 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.452507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.3.152.253:42821)
0925 10:13:06.452718 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.452720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.452727 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:13:06.452748 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.452749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.452759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.452840 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.453404 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.454064 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.454820 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.455025 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.455113 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.456596 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.456601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.456601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.456623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.458413 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.458841 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.458842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.458843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.458913 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.459147 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.459148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.459148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.459283 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.459408 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.459565 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.459736 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for oryx/127.3.152.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.252.keytab.
Entry for principal oryx/127.3.152.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.252.keytab.
I20260925 10:13:06.487260 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:32799
--webserver_interface=127.3.152.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.252.keytab
--principal=oryx/127.3.152.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:06.572005 10470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:06.572295 10470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:06.572335 10470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:06.573568 10470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:06.573662 10470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:06.573688 10470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:06.573710 10470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:06.573738 10470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:06.575402 10470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.252.keytab
--principal=oryx/127.3.152.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:32799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.152.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:06.575742 10470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:06.576009 10470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.581024 10478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.581024 10476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.581019 10475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.581269 10470 server_base.cc:1034] running on GCE node
I20260925 10:13:06.581629 10470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.581912 10470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.583097 10470 hybrid_clock.cc:648] HybridClock initialized: now 1790331186583082 us; error 46 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.585973 10470 init.cc:377] Logged in from keytab as oryx/127.3.152.252@KRBTEST.COM (short username oryx)
I20260925 10:13:06.586508 10470 webserver.cc:460] Webserver started at http://127.3.152.252:39387/ using document root <none> and password file <none>
I20260925 10:13:06.586773 10470 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.586849 10470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.587013 10470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:06.588284 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/instance:
uuid: "e9f581581e2b4c009676a895d81c973c"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "4919afb7451f26565528a888eb59f17c"
server_key_iv: "399f5bffb64c236d37c7ce8f9ea2d29a"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.588676 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal/instance:
uuid: "e9f581581e2b4c009676a895d81c973c"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "4919afb7451f26565528a888eb59f17c"
server_key_iv: "399f5bffb64c236d37c7ce8f9ea2d29a"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.590159 10470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:06.591161 10485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.591498 10470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.591614 10470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "e9f581581e2b4c009676a895d81c973c"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "4919afb7451f26565528a888eb59f17c"
server_key_iv: "399f5bffb64c236d37c7ce8f9ea2d29a"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.591712 10470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.599181 10470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.601191 10470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:06.601358 10470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.606475 10470 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.252:32799
I20260925 10:13:06.606509 10537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.252:32799 every 8 connection(s)
I20260925 10:13:06.606877 10470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
I20260925 10:13:06.607921 10538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:06.609652 10538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:13:06.613137 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10470
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.252@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
I20260925 10:13:06.621387 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.610594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.3.152.254:33181)
0925 10:13:06.610669 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.610672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.610765 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:13:06.611361 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.611363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.611377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.611434 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.612285 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.612839 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.614295 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.614507 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.614569 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.617501 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.617503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.617504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.617513 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.619500 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.620060 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.620062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.620062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.620136 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.620456 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.620476 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.620477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.620649 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.620725 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.621155 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.621256 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:13:06.621519 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.610398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46410)
0925 10:13:06.610593 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.610657 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.611227 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.611516 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.611520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.611824 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.612177 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.612183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.612962 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.612963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.614185 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.614188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.614626 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.614628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.614800 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.615583 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.615592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.617409 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.619628 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.619637 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.619639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.619920 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.620208 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.620209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.620209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.620374 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.620721 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.620728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.620941 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.621066 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.621224 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260925 10:13:06.622179 10538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:13:06.623483 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.613773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.3.152.252:32799)
0925 10:13:06.613962 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.613967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.613997 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:13:06.614018 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.614020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.614038 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.614112 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.614919 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.615943 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.616966 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.617427 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.617536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.620213 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.620220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.620220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.620228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.622175 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.622629 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.622632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.622633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.622717 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.622929 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.622929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.622930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.623102 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.623207 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.623356 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.623414 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.252@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
I20260925 10:13:06.630023 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.622536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.3.152.253:42821)
0925 10:13:06.622589 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.622591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.622601 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:06.622628 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.622629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.622644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.622701 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.623018 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.623849 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.624709 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.624995 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.625082 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.627094 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.627096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.627096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.627105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.628772 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.629169 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.629170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.629171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.629237 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.629457 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.629458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.629458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.629581 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.629661 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.629837 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.629928 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:13:06.630023 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.622450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35798)
0925 10:13:06.622502 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.622512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.622553 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.622719 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.622720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.622864 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.622938 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.622939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.624014 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.624015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.624620 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.624622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.625095 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.625096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.625162 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.625644 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.625648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.626954 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.628884 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.628885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.628886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.629073 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.629313 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.629314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.629315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.629377 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.629666 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.629667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.629714 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.629807 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.629914 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:13:06.630614 10538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.252@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
I20260925 10:13:06.637771 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.630913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.0.0.1:56652)
0925 10:13:06.630993 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.631004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.631056 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.631169 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.631170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.631297 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.631332 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.631333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.632062 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.632063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.632576 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.632586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.632990 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.632991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.633046 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.633433 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.633439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.634849 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.636637 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.636638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.636639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.636823 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.637045 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.637046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.637046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.637112 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.637394 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.637394 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.637435 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.637528 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.637637 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 10:13:06.637781 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.630953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.3.152.252:32799)
0925 10:13:06.630990 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.630991 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.631042 (+ 51us) server_negotiation.cc:407] Connection header received
0925 10:13:06.631106 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.631106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.631116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.631156 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.631397 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.631911 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.632659 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.632900 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.632955 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.634954 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.634956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.634956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.634963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.636527 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.636914 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.636915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.636915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.636973 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.637180 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.637181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.637182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.637319 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.637403 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.637565 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.637690 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":17}
I20260925 10:13:06.639393 10538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Bootstrap starting.
I20260925 10:13:06.640398 10538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:06.640707 10538 log.cc:824] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:06.641525 10538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: No bootstrap required, opened a new log
I20260925 10:13:06.643222 10538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } }
I20260925 10:13:06.643380 10538 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:06.643445 10538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f581581e2b4c009676a895d81c973c, State: Initialized, Role: FOLLOWER
I20260925 10:13:06.643617 10538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } }
I20260925 10:13:06.644014 10538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:06.643937 10543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } } }
I20260925 10:13:06.644084 10543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:13:06.645932 10554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:13:06.645993 10554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:13:06.648538 10470 master_runner.cc:428] Detected that this master e9f581581e2b4c009676a895d81c973c is joining an existing cluster
I20260925 10:13:06.648597 10470 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.152.252:32799
I20260925 10:13:06.657115 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.649597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.3.152.252:32799)
0925 10:13:06.649636 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.649638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.650027 (+ 389us) server_negotiation.cc:407] Connection header received
0925 10:13:06.650109 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.650110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.650118 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.650145 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.650791 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651550 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.652701 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.652992 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653032 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.654195 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.654196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.654196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.654203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.655863 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.656239 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.656240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.656241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.656306 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.656519 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.656520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.656521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.656670 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.656759 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.656906 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.657017 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":16}
I20260925 10:13:06.657174 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.649627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.0.0.1:56666)
0925 10:13:06.649977 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.649986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.650047 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.650204 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.650204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.650591 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.650701 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.650704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651642 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.651642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.652608 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.652610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.653125 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.653190 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.653727 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.653733 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.654093 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.655940 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.655941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.655941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.656161 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.656391 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.656392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.656392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.656450 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.656753 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.656754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.656803 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.656858 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.656946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":43,"thread_start_us":77,"threads_started":1}
I20260925 10:13:06.661742 10543 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:06.661831 10543 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } }
I20260925 10:13:06.662081 10543 leader_election.cc:288] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181), 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253:42821)
I20260925 10:13:06.662379 9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f581581e2b4c009676a895d81c973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1134c071b14f8c8882da1d9e4e9328" is_pre_election: true
I20260925 10:13:06.662374 10426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f581581e2b4c009676a895d81c973c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "108023a6f92c4f74a2a889b0749a4de3" is_pre_election: true
I20260925 10:13:06.662503 10426 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Handling vote request from an unknown peer e9f581581e2b4c009676a895d81c973c
I20260925 10:13:06.662561 10426 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate e9f581581e2b4c009676a895d81c973c for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 10:13:06.662788 10486 leader_election.cc:302] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f581581e2b4c009676a895d81c973c; no voters: 108023a6f92c4f74a2a889b0749a4de3, 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:06.662932 10543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:06.663790 10543 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 10:13:06.672317 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.649596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46422)
0925 10:13:06.649759 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.649770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.649825 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.650218 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.650219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.650588 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.650704 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.650705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651642 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.651643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.652608 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.652610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.653124 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.653190 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.668647 (+ 15457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.668652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.669024 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.671032 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.671034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.671333 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.671543 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.671544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.671603 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.672012 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.672013 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.672062 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.672131 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.672230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":56,"thread_start_us":117,"threads_started":1}
I20260925 10:13:06.672389 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.650029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.3.152.254:33181)
0925 10:13:06.650074 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.650076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.650083 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:13:06.650107 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.650108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.650118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.650167 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.650798 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651550 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.652713 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.653005 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653041 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.669159 (+ 16118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.669161 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.669161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.669170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.670926 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671421 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.671423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.671423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.671489 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671721 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.671723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.671723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.671900 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.672005 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.672186 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.672282 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:13:06.672801 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.649927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.3.152.253:42821)
0925 10:13:06.649973 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.649975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.650124 (+ 149us) server_negotiation.cc:407] Connection header received
0925 10:13:06.650234 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.650235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.650248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.650281 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.650853 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651680 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.652884 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.653216 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653263 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.669258 (+ 15995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.669260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.669261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.669270 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.671065 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671512 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.671514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.671515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.671583 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671774 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.671776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.671776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.671908 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.672019 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.672626 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.672713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:06.673269 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.649877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35804)
0925 10:13:06.650058 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.650069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.650154 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.650350 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.650351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.650656 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.650752 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.650754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.651800 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.651801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.652775 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.652778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.653347 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.653348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.653415 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.668598 (+ 15183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.668605 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.669146 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.671153 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.671154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.671366 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.671658 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.671658 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.671659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.671719 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.672432 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.672433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.672486 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.673078 (+ 592us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.673192 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 10:13:06.675951 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9880
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331185, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.254@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
I20260925 10:13:06.683187 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.674833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.3.152.252:32799)
0925 10:13:06.674895 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.674898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.674906 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:13:06.674941 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.674942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.674953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.675028 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.675275 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.676145 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.677270 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.677621 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.677696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.680155 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.680157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.680157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.680164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.681878 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.682302 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.682303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.682303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.682368 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.682586 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.682586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.682587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.682744 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.682818 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.682999 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.683083 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:13:06.683457 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:37685
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:33181
--webserver_port=37739
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:13:06.683216 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.674751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.0.0.1:56676)
0925 10:13:06.674804 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.674816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.674858 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.675029 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.675029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.675156 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.675198 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.675199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.676251 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.676252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.677186 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.677189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.677710 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.677711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.677797 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.678241 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.678246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.680046 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.681970 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.681971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.681971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.682214 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.682443 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.682444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.682445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.682516 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.682818 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.682819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.682871 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.682945 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.683065 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:13:06.685359 9818 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.152.252:32799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e9f581581e2b4c009676a895d81c973c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.152.252"
port: 32799
}
attrs {
promote: true
}
}
dest_uuid: "2a1134c071b14f8c8882da1d9e4e9328"
cas_config_opid_index: 5
I20260925 10:13:06.685635 9818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:06.686036 10377 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e9f581581e2b4c009676a895d81c973c (127.3.152.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.686352 10507 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 FOLLOWER]: Refusing update from remote peer 2a1134c071b14f8c8882da1d9e4e9328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 10:13:06.686604 10378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:06.687152 10377 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.152.252:32799
I20260925 10:13:06.687237 10377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.687315 10377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:06.688088 10507 raft_consensus.cc:858] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
W20260925 10:13:06.688169 10507 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 32799 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:06.688686 10507 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.690056 10543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1134c071b14f8c8882da1d9e4e9328. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.690132 10543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:13:06.690168 10543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.690207 10543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:13:06.690234 10543 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } }
I20260925 10:13:06.690274 10543 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 10:13:06.691082 10470 master.cc:537] Master@127.3.152.252:32799 shutting down...
I20260925 10:13:06.691834 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.691953 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:06.692622 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.686109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.0.0.1:35820)
0925 10:13:06.687430 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.687441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.687488 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.688114 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.688115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.688117 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:06.688182 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.688184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.689539 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.689540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.690953 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.690957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.692300 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.692301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.692383 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.692415 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.692476 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.692523 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302}
I20260925 10:13:06.692878 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.687720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.3.152.253:42821)
0925 10:13:06.687764 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.687766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.687775 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:06.687825 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.687825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.687851 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.687919 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:06.688348 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.689183 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.691289 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.692079 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.692154 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.692258 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.692731 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.692788 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:13:06.693109 10426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Refusing update from remote peer 2a1134c071b14f8c8882da1d9e4e9328: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260925 10:13:06.693310 9805 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 -> Peer e9f581581e2b4c009676a895d81c973c (127.3.152.252:32799): Couldn't send request to peer e9f581581e2b4c009676a895d81c973c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 10:13:06.693339 10377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 10:13:06.693920 10426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e9f581581e2b4c009676a895d81c973c (127.3.152.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.694846 10470 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 10:13:06.694948 10470 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 10:13:06.694995 10470 tablet_replica.cc:330] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: stopping tablet replica
I20260925 10:13:06.695331 10569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.695549 10569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:13:06.695551 10455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a1134c071b14f8c8882da1d9e4e9328. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: NON_VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: true } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.695713 10455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: This master's current role is: LEARNER
I20260925 10:13:06.700562 10377 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: attempting to promote NON_VOTER 108023a6f92c4f74a2a889b0749a4de3 to VOTER
I20260925 10:13:06.700790 10377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:06.701270 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.701142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.0.0.1:56692)
0925 10:13:06.701184 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.701240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:32799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 10:13:06.701503 10426 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 LEARNER]: Refusing update from remote peer 2a1134c071b14f8c8882da1d9e4e9328: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 10:13:06.701686 10378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 10:13:06.701978 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.701824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.0.0.1:56706)
0925 10:13:06.701930 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.701953 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:32799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 10:13:06.702181 9805 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 -> Peer e9f581581e2b4c009676a895d81c973c (127.3.152.252:32799): Couldn't send request to peer e9f581581e2b4c009676a895d81c973c. Status: Network error: Client connection negotiation failed: client connection to 127.3.152.252:32799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 10:13:06.702557 10378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.703517 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.703591 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:06.703648 10378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.703701 10378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:06.704243 10426 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 108023a6f92c4f74a2a889b0749a4de3 (127.3.152.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.704659 10470 master.cc:552] Master@127.3.152.252:32799 shutdown complete.
I20260925 10:13:06.704696 10470 master_runner.cc:305] Clearing existing system tablet
I20260925 10:13:06.704829 10470 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.704866 10470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.705222 10455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a1134c071b14f8c8882da1d9e4e9328" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.705305 10455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:13:06.708279 10470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 10:13:06.708937 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.709088 10470 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.709157 10470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:06.709185 10470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "e9f581581e2b4c009676a895d81c973c"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "4919afb7451f26565528a888eb59f17c"
server_key_iv: "399f5bffb64c236d37c7ce8f9ea2d29a"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.710005 10470 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:06.711759 10470 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 10:13:06.711839 10470 log.cc:1196] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 10:13:06.712142 10470 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Deleting consensus metadata
I20260925 10:13:06.712280 10470 master_runner.cc:315] Copying system tablet from 127.3.152.254:33181
I20260925 10:13:06.712944 10470 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.152.254:33181
I20260925 10:13:06.720930 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.713321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.3.152.254:33181)
0925 10:13:06.713376 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.713378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.713524 (+ 146us) server_negotiation.cc:407] Connection header received
0925 10:13:06.713582 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.713583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.713596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.713665 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.714138 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.714908 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.716193 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.716675 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.716759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.717968 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.717970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.717970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.717978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.719662 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.720084 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.720085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.720085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.720141 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.720340 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.720341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.720341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.720503 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.720549 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.720757 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.720836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:13:06.721381 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.713249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46434)
0925 10:13:06.713427 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.713440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.713518 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.713671 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.713672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.713934 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.713999 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.714001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.715007 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.715008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.716090 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.716094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.716822 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.716823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.716892 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.717471 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.717476 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.717865 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.719763 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.719764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.719764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.720003 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.720209 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.720209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.720210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.720274 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.720598 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.720599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.720646 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.721128 (+ 482us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.721255 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260925 10:13:06.721735 9838 tablet_copy_service.cc:140] P 2a1134c071b14f8c8882da1d9e4e9328: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e9f581581e2b4c009676a895d81c973c ({username='oryx', principal='oryx/127.3.152.252@KRBTEST.COM'} at 127.0.0.1:46434)
I20260925 10:13:06.721794 9838 tablet_copy_service.cc:161] P 2a1134c071b14f8c8882da1d9e4e9328: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e9f581581e2b4c009676a895d81c973c at {username='oryx', principal='oryx/127.3.152.252@KRBTEST.COM'} at 127.0.0.1:46434: session id = e9f581581e2b4c009676a895d81c973c-00000000000000000000000000000000
I20260925 10:13:06.722476 9838 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 10:13:06.723078 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 10:13:06.724483 10470 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: tablet copy: Starting download of 0 data blocks...
I20260925 10:13:06.724637 10470 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: tablet copy: Starting download of 1 WAL segments...
I20260925 10:13:06.725401 10470 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 10:13:06.726382 9838 tablet_copy_service.cc:342] P 2a1134c071b14f8c8882da1d9e4e9328: Request end of tablet copy session e9f581581e2b4c009676a895d81c973c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.152.252@KRBTEST.COM'} at 127.0.0.1:46434
I20260925 10:13:06.726441 9838 tablet_copy_service.cc:433] P 2a1134c071b14f8c8882da1d9e4e9328: ending tablet copy session e9f581581e2b4c009676a895d81c973c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e9f581581e2b4c009676a895d81c973c
I20260925 10:13:06.729708 10470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.730947 10585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.731067 10582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.731168 10470 server_base.cc:1034] running on GCE node
W20260925 10:13:06.737807 10583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.738036 10470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.738234 10470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.739385 10470 hybrid_clock.cc:648] HybridClock initialized: now 1790331186739290 us; error 116 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.741118 10470 init.cc:377] Logged in from keytab as oryx/127.3.152.252@KRBTEST.COM (short username oryx)
I20260925 10:13:06.741402 10470 webserver.cc:460] Webserver started at http://127.3.152.252:34461/ using document root <none> and password file <none>
I20260925 10:13:06.741532 10470 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.741570 10470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.742412 10470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.743135 10592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.743345 10470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:13:06.743399 10470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "e9f581581e2b4c009676a895d81c973c"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "4919afb7451f26565528a888eb59f17c"
server_key_iv: "399f5bffb64c236d37c7ce8f9ea2d29a"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.743458 10470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.767400 10470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.769793 10470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.772965 10470 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.252:32799
I20260925 10:13:06.773186 10644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.252:32799 every 8 connection(s)
I20260925 10:13:06.773536 10470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
I20260925 10:13:06.774598 10645 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:13:06.775146 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Bootstrap starting.
I20260925 10:13:06.777784 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:13:06.778111 10645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: Bootstrap complete.
I20260925 10:13:06.778378 10645 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:06.778465 10645 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9f581581e2b4c009676a895d81c973c, State: Initialized, Role: LEARNER
I20260925 10:13:06.778501 10645 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:06.778676 10645 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:06.778751 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } } }
I20260925 10:13:06.778954 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c [sys.catalog]: This master's current role is: LEARNER
I20260925 10:13:06.781510 10659 catalog_manager.cc:1259] Loaded cluster ID: 60234a734d98485e90f81b758c7f85cb
I20260925 10:13:06.781605 10659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: loading cluster ID for follower catalog manager: success
I20260925 10:13:06.782856 10659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: acquiring CA information for follower catalog manager: success
I20260925 10:13:06.783274 10659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f581581e2b4c009676a895d81c973c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 10:13:06.798476 10563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:06.798750 10563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:06.798799 10563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:06.800388 10563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:06.800459 10563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:06.800513 10563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:06.802217 10563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:37685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=37739
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:06.802525 10563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:06.802793 10563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.809257 10664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.809289 10665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.809432 10667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.809696 10563 server_base.cc:1034] running on GCE node
I20260925 10:13:06.809898 10563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.810160 10563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.811326 10563 hybrid_clock.cc:648] HybridClock initialized: now 1790331186811304 us; error 37 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.813937 10563 init.cc:377] Logged in from keytab as oryx/127.3.152.193@KRBTEST.COM (short username oryx)
I20260925 10:13:06.814507 10563 webserver.cc:460] Webserver started at http://127.3.152.193:37739/ using document root <none> and password file <none>
I20260925 10:13:06.814755 10563 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.814838 10563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.816535 10563 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:06.817262 10674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.817447 10563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.817543 10563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "8884c173f64d429c8e8d1976af0804bd"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "e0699ce126eda230d5adde0d3a51931a"
server_key_iv: "c190a1e7a8ebf79fee783494a9b869b6"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.817929 10563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.830320 10563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.832811 10563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:06.832980 10563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.833227 10563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:06.833544 10563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:06.833598 10563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.833645 10563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:06.833668 10563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.839509 10563 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:37685
I20260925 10:13:06.839551 10787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:37685 every 8 connection(s)
I20260925 10:13:06.840013 10563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
I20260925 10:13:06.850274 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10563
I20260925 10:13:06.850451 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10014
I20260925 10:13:06.851522 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.841294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:51285 (local address 127.3.152.253:42821)
0925 10:13:06.841342 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.841344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.841352 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:13:06.842178 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.842179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.842192 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.842225 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.843070 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.843695 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.844998 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.845212 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.845299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.848010 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.848012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.848012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.848021 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.849816 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.850410 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.850412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.850412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.850471 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.850740 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.850741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.850741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.850881 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.850920 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.851293 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.851380 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:13:06.852109 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.841107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.3.152.193:51285)
0925 10:13:06.841219 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.841229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.842072 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.842316 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.842317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.842725 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.842986 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.842991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.843796 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.843797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.844909 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.844911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.845361 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.845362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.845475 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.845991 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.845999 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.847848 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.849914 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.849922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.849925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.850318 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.850546 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.850546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.850547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.850674 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.850961 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.850967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.851140 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.851612 (+ 472us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.851803 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":31,"thread_start_us":54,"threads_started":1}
I20260925 10:13:06.852751 10794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.840820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.3.152.193:49203)
0925 10:13:06.841067 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.841076 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.842069 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.842389 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.842390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.842766 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.842986 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.842991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.844965 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.844966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.846278 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.846281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.846966 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.846967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.847031 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.847440 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.847444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.848753 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.851521 (+ 2768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.851522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.851523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.851756 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.851954 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.851954 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.851955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.852049 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.852341 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.852342 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.852384 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.852458 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.852572 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":26,"thread_start_us":106,"threads_started":1}
I20260925 10:13:06.852938 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.841224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40011 (local address 127.3.152.254:33181)
0925 10:13:06.841271 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.841273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.841283 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:06.842160 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.842162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.842175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.842240 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.843133 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.843804 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.845142 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.845459 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.845499 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.848329 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.848330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.848330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.848338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.849936 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.850464 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.850465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.850466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.850523 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.852095 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.852096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.852096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.852233 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.852283 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.852747 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.852829 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:13:06.853127 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.840825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:49203 (local address 127.3.152.252:32799)
0925 10:13:06.840970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:06.840972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:06.841143 (+ 171us) server_negotiation.cc:407] Connection header received
0925 10:13:06.842250 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:06.842251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:06.842266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:06.842317 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:06.843774 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.844874 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.846380 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.846841 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.846889 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.848841 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:06.848842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:06.848842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:06.848849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:06.851432 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.851832 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.851833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.851833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.851888 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:06.852116 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:06.852117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:06.852118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:06.852259 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:06.852317 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:06.852881 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 10:13:06.852982 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260925 10:13:06.853449 10793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:06.840821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:40011)
0925 10:13:06.840977 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:06.841022 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:06.842043 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:06.842312 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:06.842315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:06.842712 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:06.843006 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.843008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.843891 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.843892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.845042 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:06.845043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:06.845591 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:06.845592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:06.845660 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:06.846173 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:06.846178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:06.848252 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:06.850015 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.850015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.850016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.850318 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.851928 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:06.851928 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:06.851929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:06.852014 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:06.852599 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:06.852600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:06.852633 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 10:13:06.852703 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:13:06.853147 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":72,"thread_start_us":88,"threads_started":1}
I20260925 10:13:06.853875 10789 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:06.853986 10789 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:06.854189 10789 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:06.854619 9818 ts_manager.cc:194] Re-registered known tserver with Master: 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685)
I20260925 10:13:06.855041 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:40011
I20260925 10:13:06.856382 10788 heartbeater.cc:347] Connected to a master server at 127.3.152.252:32799
I20260925 10:13:06.856455 10788 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:06.856534 10788 heartbeater.cc:511] Master 127.3.152.252:32799 requested a full tablet report, sending...
I20260925 10:13:06.856945 10609 ts_manager.cc:194] Registered new tserver with Master: 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685)
I20260925 10:13:06.856992 10790 heartbeater.cc:347] Connected to a master server at 127.3.152.253:42821
I20260925 10:13:06.857040 10790 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:06.857101 10790 heartbeater.cc:511] Master 127.3.152.253:42821 requested a full tablet report, sending...
I20260925 10:13:06.857249 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:37495
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:33181
--webserver_port=36381
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:13:06.857486 10416 ts_manager.cc:194] Registered new tserver with Master: 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685)
W20260925 10:13:06.940562 10796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:06.940819 10796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:06.940861 10796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:06.942121 10796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:06.942219 10796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:06.942268 10796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:06.944278 10796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:37495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=36381
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:06.944602 10796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:06.944854 10796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:06.950645 10802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.950645 10801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:06.950687 10804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:06.950938 10796 server_base.cc:1034] running on GCE node
I20260925 10:13:06.951131 10796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:06.951367 10796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:06.952505 10796 hybrid_clock.cc:648] HybridClock initialized: now 1790331186952480 us; error 39 us; skew 500 ppm
Sep 25 10:13:06 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:06.955060 10796 init.cc:377] Logged in from keytab as oryx/127.3.152.194@KRBTEST.COM (short username oryx)
I20260925 10:13:06.955546 10796 webserver.cc:460] Webserver started at http://127.3.152.194:36381/ using document root <none> and password file <none>
I20260925 10:13:06.955761 10796 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:06.955814 10796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:06.957247 10796 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.957964 10811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.958189 10796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:06.958303 10796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "3f50cb869fa94bc69e439a6a12821286"
format_stamp: "Formatted at 2026-09-25 10:13:05 on dist-test-slave-dzwq"
server_key: "c03388c9e2f4d050c0acbafe1d91100f"
server_key_iv: "fca95c350b40e3dece272e7625dc59d9"
server_key_version: "encryptionkey@0"
I20260925 10:13:06.958673 10796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:06.990458 10796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:06.992847 10796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:06.993045 10796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:06.993319 10796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:06.993675 10796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:06.993747 10796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:06.993788 10796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:06.993816 10796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:07.000236 10796 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:37495
I20260925 10:13:07.000289 10924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:37495 every 8 connection(s)
I20260925 10:13:07.000658 10796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:07.002588 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10796
I20260925 10:13:07.002768 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10148
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
I20260925 10:13:07.010308 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:34029
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:33181
--webserver_port=45583
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331186, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
I20260925 10:13:07.014226 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42415 (local address 127.3.152.254:33181)
0925 10:13:07.001922 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.001924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.001938 (+ 14us) server_negotiation.cc:407] Connection header received
0925 10:13:07.002514 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.002516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.002530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.002563 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.003923 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.004742 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.006427 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.006742 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.006784 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.009865 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.009867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.009867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.009875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.012343 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.012971 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.012973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.012973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.013038 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.013384 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.013385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.013386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.013534 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.013602 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.013973 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.014064 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:13:07.014429 10931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:42415)
0925 10:13:07.001891 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.001901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.002382 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.002765 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.002769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.003401 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.003803 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.003810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.004881 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.004882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.006315 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.006318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.006875 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.006875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.007027 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.007877 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.007888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.009727 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.012444 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.012453 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.012455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.012878 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.013127 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.013128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.013128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.013288 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.013609 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.013617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.013803 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.013929 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.014158 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":197,"thread_start_us":70,"threads_started":1}
I20260925 10:13:07.014673 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.3.152.194:47473)
0925 10:13:07.001744 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.001776 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.002382 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.002765 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.002769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.003761 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.003813 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.003814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.004881 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.004882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.007044 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.007046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.007566 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.007639 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.008783 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.008788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.010249 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.012636 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.012637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.012638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.012912 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.013377 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.013377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.013377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.013440 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.013754 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.013755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.013803 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.013929 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.014205 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1335,"thread_start_us":112,"threads_started":1}
I20260925 10:13:07.014945 10926 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:07.015034 10926 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.015185 10926 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:07.015612 9818 ts_manager.cc:194] Re-registered known tserver with Master: 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
I20260925 10:13:07.016111 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:42415
I20260925 10:13:07.016724 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:47473 (local address 127.3.152.253:42821)
0925 10:13:07.001580 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.001582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.001833 (+ 251us) server_negotiation.cc:407] Connection header received
0925 10:13:07.002514 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.002516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.002530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.002563 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.003923 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.004732 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007146 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.007468 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007557 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.010363 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.010365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.010366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.010375 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.012545 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.013226 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.013227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.013228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.013279 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.013520 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.013522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.013522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.013675 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.013755 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.016524 (+ 2769us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.016631 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:13:07.017089 10927 heartbeater.cc:347] Connected to a master server at 127.3.152.253:42821
I20260925 10:13:07.017135 10927 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.017198 10927 heartbeater.cc:511] Master 127.3.152.253:42821 requested a full tablet report, sending...
I20260925 10:13:07.017575 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.3.152.194:51901)
0925 10:13:07.001663 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.001713 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.002404 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.003184 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.003186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.003440 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.003802 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.003809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.005246 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.005252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.006315 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.006318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.007571 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.007633 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.009922 (+ 2289us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.009928 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.011373 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.016137 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.016138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.016139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.016372 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.016686 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.016687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.016687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.016757 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.017108 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.017109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.017158 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.017246 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.017413 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":14,"thread_start_us":83,"threads_started":1}
I20260925 10:13:07.017630 10416 ts_manager.cc:194] Registered new tserver with Master: 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
I20260925 10:13:07.018043 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.001502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:51901 (local address 127.3.152.252:32799)
0925 10:13:07.001695 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.001696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.001805 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:13:07.002532 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.002533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.002635 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.002682 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.003931 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.004938 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007159 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.007440 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.007474 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.011473 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.011475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.011475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.011484 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.013915 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.016479 (+ 2564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.016481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.016481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.016558 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.016835 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.016836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.016836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.016996 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.017066 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.017790 (+ 724us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.017905 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260925 10:13:07.018535 10925 heartbeater.cc:347] Connected to a master server at 127.3.152.252:32799
I20260925 10:13:07.018591 10925 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.018640 10925 heartbeater.cc:511] Master 127.3.152.252:32799 requested a full tablet report, sending...
I20260925 10:13:07.018929 10609 ts_manager.cc:194] Registered new tserver with Master: 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
W20260925 10:13:07.097819 10932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:07.098071 10932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:07.098097 10932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:07.099340 10932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:07.099390 10932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:07.099419 10932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:07.100983 10932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:34029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=45583
--tserver_master_addrs=127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:07.101246 10932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:07.101521 10932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:07.107179 10940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:07.107204 10938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:07.107255 10932 server_base.cc:1034] running on GCE node
W20260925 10:13:07.107204 10937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:07.107645 10932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:07.107914 10932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:07.109113 10932 hybrid_clock.cc:648] HybridClock initialized: now 1790331187109104 us; error 58 us; skew 500 ppm
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331187, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:07.111716 10932 init.cc:377] Logged in from keytab as oryx/127.3.152.195@KRBTEST.COM (short username oryx)
I20260925 10:13:07.112228 10932 webserver.cc:460] Webserver started at http://127.3.152.195:45583/ using document root <none> and password file <none>
I20260925 10:13:07.112422 10932 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:07.112474 10932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:07.113821 10932 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:07.114378 10947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:07.114568 10932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:07.114645 10932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "65845fb86457409cbf24d3143cbe5494"
format_stamp: "Formatted at 2026-09-25 10:13:06 on dist-test-slave-dzwq"
server_key: "7526d24b3a63bc35daa35054f536ef9d"
server_key_iv: "0c23dfa0bee959701aa2268dd25283d7"
server_key_version: "encryptionkey@0"
I20260925 10:13:07.115087 10932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:07.133745 10932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:07.136288 10932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:07.136463 10932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:07.136750 10932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:07.137094 10932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:07.137128 10932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:07.137172 10932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:07.137208 10932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:07.143457 10932 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:34029
I20260925 10:13:07.143533 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:34029 every 8 connection(s)
I20260925 10:13:07.143924 10932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:07.147657 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 10932
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331187, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.253@KRBTEST.COM
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331187, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.252@KRBTEST.COM
Sep 25 10:13:07 dist-test-slave-dzwq krb5kdc[9773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331187, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:07.157632 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.144976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:42821 (local address 127.3.152.195:35747)
0925 10:13:07.145180 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.145191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.146110 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.146444 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.146449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.147032 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.147291 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.147297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148343 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.148344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.149753 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.149755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150303 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.150455 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.151256 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.151261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.153863 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.155715 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.155723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.155726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.156153 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.156381 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.156382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.156382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.156524 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.156855 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.156863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.157036 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.157155 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.157355 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":223,"thread_start_us":79,"threads_started":1}
I20260925 10:13:07.157984 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.145101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:35747 (local address 127.3.152.253:42821)
0925 10:13:07.145151 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.145154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.145250 (+ 96us) server_negotiation.cc:407] Connection header received
0925 10:13:07.146236 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.146238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.146252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.146285 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.147403 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148209 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.149860 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150167 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150209 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.153945 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.153947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.153947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.153954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.155632 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.156243 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.156244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.156245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.156305 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.156606 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.156607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.156607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.156750 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.156827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.157794 (+ 967us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.157873 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:07.158335 11063 heartbeater.cc:347] Connected to a master server at 127.3.152.253:42821
I20260925 10:13:07.158435 11063 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.158633 11063 heartbeater.cc:511] Master 127.3.152.253:42821 requested a full tablet report, sending...
I20260925 10:13:07.159256 10416 ts_manager.cc:194] Registered new tserver with Master: 65845fb86457409cbf24d3143cbe5494 (127.3.152.195:34029)
I20260925 10:13:07.159849 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.144692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:33367 (local address 127.3.152.254:33181)
0925 10:13:07.144747 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.144750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.144973 (+ 223us) server_negotiation.cc:407] Connection header received
0925 10:13:07.146238 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.146239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.146252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.146377 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.147388 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148247 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.149856 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150191 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150248 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.155375 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.155376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.155377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.155383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.158013 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.158935 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.158936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.158937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.159000 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.159248 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.159249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.159249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.159400 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.159445 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.159662 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.159742 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:07.159881 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.144709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:57135 (local address 127.3.152.252:32799)
0925 10:13:07.145114 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.145116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.145360 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:13:07.146423 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.146424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.146436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.146501 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.147403 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148209 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.149859 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150167 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150209 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.155990 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.155991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.155992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.155999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.158190 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.158961 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.158962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.158962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.159019 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.159282 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.159282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.159283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.159415 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.159459 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.159686 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.159783 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260925 10:13:07.160223 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.144698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:33367)
0925 10:13:07.144851 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.144886 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.146099 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.146495 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.146497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.147008 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.147291 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.147296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148676 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.148677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.149755 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.149758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150333 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.150456 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.151192 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.151202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.155289 (+ 4087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.158364 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.158365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.158365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.158818 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.159096 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.159097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.159097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.159174 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.159492 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.159492 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.159536 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.159998 (+ 462us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.160108 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":19,"thread_start_us":78,"threads_started":1}
I20260925 10:13:07.160554 11062 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33181
I20260925 10:13:07.160612 11062 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.160743 11062 heartbeater.cc:511] Master 127.3.152.254:33181 requested a full tablet report, sending...
I20260925 10:13:07.161008 9818 ts_manager.cc:194] Re-registered known tserver with Master: 65845fb86457409cbf24d3143cbe5494 (127.3.152.195:34029)
I20260925 10:13:07.161484 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.150846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.3.152.252:32799)
0925 10:13:07.150985 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.150987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.152642 (+ 1655us) server_negotiation.cc:407] Connection header received
0925 10:13:07.152749 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.152750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.152764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.152807 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.153520 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.154260 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.155282 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.155591 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.155631 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.157113 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.157115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.157116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.157122 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.159660 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.160368 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.160369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.160369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.160427 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.160713 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.160714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.160714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.160875 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.160963 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.161261 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.161341 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260925 10:13:07.161509 9818 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:33367
I20260925 10:13:07.162063 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.148686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.3.152.254:33181)
0925 10:13:07.149468 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.149470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.149476 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:13:07.149495 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.149496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.149508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.149564 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.150351 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.151058 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.152129 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.152441 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.152512 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.154203 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.154204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.154204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.154211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.160351 (+ 6140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.160737 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.160738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.160738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.160805 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.161040 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.161041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.161041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.161182 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.161274 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.161603 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.161689 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":101,"threads_started":1}
I20260925 10:13:07.160539 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.145144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:32799 (local address 127.3.152.195:57135)
0925 10:13:07.145301 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.145311 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.146108 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.146561 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.146562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.147031 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:07.147291 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.147297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.148343 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.148344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.149753 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.149755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.150306 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.150306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.150455 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.151382 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:07.151387 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:07.155861 (+ 4474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:07.158371 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.158372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.158372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.158620 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.159098 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:07.159099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:07.159099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:07.159208 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:07.159524 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:07.159525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:07.159570 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.160193 (+ 623us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.160411 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":73,"thread_start_us":98,"threads_started":1}
I20260925 10:13:07.163056 11061 heartbeater.cc:347] Connected to a master server at 127.3.152.252:32799
I20260925 10:13:07.163106 11061 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:07.163177 11061 heartbeater.cc:511] Master 127.3.152.252:32799 requested a full tablet report, sending...
I20260925 10:13:07.163440 10609 ts_manager.cc:194] Registered new tserver with Master: 65845fb86457409cbf24d3143cbe5494 (127.3.152.195:34029)
I20260925 10:13:07.164604 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.149068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.3.152.253:42821)
0925 10:13:07.149275 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.149277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.151169 (+ 1892us) server_negotiation.cc:407] Connection header received
0925 10:13:07.151408 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.151409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.151422 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.151463 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.152275 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.153230 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.154586 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.155379 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.155428 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.161526 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.161527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.161527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.161534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.163388 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.163835 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.163836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.163836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.163904 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.164140 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.164140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.164141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.164271 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.164373 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.164439 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.164516 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":38,"threads_started":1}
I20260925 10:13:07.164001 9818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:07.165956 9818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:07.173727 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.170458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.3.152.194:37495)
0925 10:13:07.170695 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.170698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.170707 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:07.170741 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.170743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.170758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.170953 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:07.171008 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.171697 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.172873 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.173394 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.173486 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.173571 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.173605 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.173639 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260925 10:13:07.173727 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.169984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:37495 (local address 127.0.0.1:33754)
0925 10:13:07.170175 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.170187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.170229 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.170857 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.170858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.170859 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:07.170886 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.170887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.171795 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.171796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.172790 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.172792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.173484 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.173485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.173541 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.173544 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.173592 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.173638 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260925 10:13:07.174495 10859 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c (DEFAULT_TABLE table=test-table [id=2cfa280ac0aa484c98379a28db2550a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:07.174649 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.170054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:34029 (local address 127.0.0.1:45138)
0925 10:13:07.170177 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.170187 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.170229 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.170829 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.170830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.170832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:07.170881 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.170883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.172418 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.172418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.173490 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.173492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.174379 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.174380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.174446 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.174451 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.174520 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.174581 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260925 10:13:07.174672 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.170249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.3.152.195:34029)
0925 10:13:07.170646 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.170650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.170660 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:07.170702 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.170704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.170721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.170812 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:07.170943 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.172330 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.173582 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.174291 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.174367 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.174435 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.174534 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.174574 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":109,"threads_started":1}
I20260925 10:13:07.174918 10859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:07.175249 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.169955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37685 (local address 127.0.0.1:32790)
0925 10:13:07.170132 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.170145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.170199 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.170718 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.170719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.170722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:07.170785 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.170788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.172198 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.172199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.173419 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.173421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.175029 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.175030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.175096 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.175100 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.175137 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.175175 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":19}
I20260925 10:13:07.175350 10995 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c (DEFAULT_TABLE table=test-table [id=2cfa280ac0aa484c98379a28db2550a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:07.175684 10995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:07.175837 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.170355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.3.152.193:37685)
0925 10:13:07.170527 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.170533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.170544 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:07.170588 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.170590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.170608 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.170703 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:07.170874 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.172100 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.173531 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.174607 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.175552 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.175617 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.175644 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.175691 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260925 10:13:07.176664 10722 tablet_service.cc:1467] Processing CreateTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c (DEFAULT_TABLE table=test-table [id=2cfa280ac0aa484c98379a28db2550a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:07.176980 10722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:07.178224 11087 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Bootstrap starting.
I20260925 10:13:07.178776 11087 tablet_bootstrap.cc:654] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:07.178998 11087 log.cc:824] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:07.179625 11089 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Bootstrap starting.
I20260925 10:13:07.179800 11087 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: No bootstrap required, opened a new log
I20260925 10:13:07.179864 11087 ts_tablet_manager.cc:1397] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:07.180326 11089 tablet_bootstrap.cc:654] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:07.180511 11090 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Bootstrap starting.
I20260925 10:13:07.180769 11089 log.cc:824] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:07.181172 11090 tablet_bootstrap.cc:654] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:07.181505 11090 log.cc:824] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:07.181697 11089 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: No bootstrap required, opened a new log
I20260925 10:13:07.181782 11089 ts_tablet_manager.cc:1397] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:13:07.182273 11090 tablet_bootstrap.cc:492] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: No bootstrap required, opened a new log
I20260925 10:13:07.182361 11090 ts_tablet_manager.cc:1397] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:07.182641 11087 raft_consensus.cc:348] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.182837 11087 raft_consensus.cc:374] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:07.182873 11087 raft_consensus.cc:729] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f50cb869fa94bc69e439a6a12821286, State: Initialized, Role: FOLLOWER
I20260925 10:13:07.182998 11087 consensus_queue.cc:260] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.183264 11087 ts_tablet_manager.cc:1428] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:13:07.183386 10926 heartbeater.cc:503] Master 127.3.152.254:33181 was elected leader, sending a full tablet report...
I20260925 10:13:07.184710 11089 raft_consensus.cc:348] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.184845 11089 raft_consensus.cc:374] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:07.184868 11089 raft_consensus.cc:729] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65845fb86457409cbf24d3143cbe5494, State: Initialized, Role: FOLLOWER
I20260925 10:13:07.185003 11089 consensus_queue.cc:260] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.185273 11089 ts_tablet_manager.cc:1428] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:13:07.185318 11062 heartbeater.cc:503] Master 127.3.152.254:33181 was elected leader, sending a full tablet report...
I20260925 10:13:07.185323 11090 raft_consensus.cc:348] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.185464 11090 raft_consensus.cc:374] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:07.185504 11090 raft_consensus.cc:729] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8884c173f64d429c8e8d1976af0804bd, State: Initialized, Role: FOLLOWER
I20260925 10:13:07.185639 11090 consensus_queue.cc:260] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.185902 11090 ts_tablet_manager.cc:1428] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:13:07.186255 10789 heartbeater.cc:503] Master 127.3.152.254:33181 was elected leader, sending a full tablet report...
I20260925 10:13:07.190305 11094 raft_consensus.cc:482] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:07.190395 11094 raft_consensus.cc:504] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.190764 11094 leader_election.cc:288] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685), 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
I20260925 10:13:07.194768 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.191171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53315 (local address 127.3.152.194:37495)
0925 10:13:07.191215 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.191217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.191224 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:13:07.191257 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.191257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.191265 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.191316 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:07.191454 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.192212 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.193741 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.194355 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.194422 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.194431 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.194648 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.194697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:13:07.194768 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.190922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:37495 (local address 127.3.152.195:53315)
0925 10:13:07.190985 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.190999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.191040 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.191333 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.191333 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.191335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:07.191380 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.191381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.192309 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.192311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.193633 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.193637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.194459 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.194460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.194553 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.194559 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.194633 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.194682 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":78}
I20260925 10:13:07.194839 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.191116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:40543 (local address 127.3.152.193:37685)
0925 10:13:07.191157 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.191159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.191168 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:07.191191 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.191192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.191202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.191258 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:07.191443 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.192212 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.193798 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.194336 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.194422 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.194431 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.194716 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.194771 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:13:07.195123 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.190945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37685 (local address 127.3.152.195:40543)
0925 10:13:07.191006 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:07.191014 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:07.191046 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:07.191266 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:07.191267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:07.191269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:07.191351 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.191354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.192309 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.192311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.193710 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:07.193712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.194459 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.194460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:07.194553 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.194559 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:13:07.194706 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 10:13:07.194748 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":55}
I20260925 10:13:07.195194 10879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3fdb48b914b2aab1a5a32fe36ca2c" candidate_uuid: "65845fb86457409cbf24d3143cbe5494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f50cb869fa94bc69e439a6a12821286" is_pre_election: true
I20260925 10:13:07.195349 10879 raft_consensus.cc:2454] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65845fb86457409cbf24d3143cbe5494 in term 0.
I20260925 10:13:07.195547 10951 leader_election.cc:302] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f50cb869fa94bc69e439a6a12821286, 65845fb86457409cbf24d3143cbe5494; no voters:
I20260925 10:13:07.195753 11094 raft_consensus.cc:2781] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:07.195719 10742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3fdb48b914b2aab1a5a32fe36ca2c" candidate_uuid: "65845fb86457409cbf24d3143cbe5494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8884c173f64d429c8e8d1976af0804bd" is_pre_election: true
I20260925 10:13:07.195832 11094 raft_consensus.cc:482] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:07.195868 11094 raft_consensus.cc:3037] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:07.195892 10742 raft_consensus.cc:2454] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65845fb86457409cbf24d3143cbe5494 in term 0.
I20260925 10:13:07.196799 11094 raft_consensus.cc:504] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.196950 11094 leader_election.cc:288] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [CANDIDATE]: Term 1 election: Requested vote from peers 8884c173f64d429c8e8d1976af0804bd (127.3.152.193:37685), 3f50cb869fa94bc69e439a6a12821286 (127.3.152.194:37495)
I20260925 10:13:07.197091 10879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3fdb48b914b2aab1a5a32fe36ca2c" candidate_uuid: "65845fb86457409cbf24d3143cbe5494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f50cb869fa94bc69e439a6a12821286"
I20260925 10:13:07.197115 10742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ca3fdb48b914b2aab1a5a32fe36ca2c" candidate_uuid: "65845fb86457409cbf24d3143cbe5494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8884c173f64d429c8e8d1976af0804bd"
I20260925 10:13:07.197175 10879 raft_consensus.cc:3037] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:07.197216 10742 raft_consensus.cc:3037] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:07.198094 10742 raft_consensus.cc:2454] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65845fb86457409cbf24d3143cbe5494 in term 1.
I20260925 10:13:07.198153 10879 raft_consensus.cc:2454] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65845fb86457409cbf24d3143cbe5494 in term 1.
I20260925 10:13:07.198325 10949 leader_election.cc:302] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65845fb86457409cbf24d3143cbe5494, 8884c173f64d429c8e8d1976af0804bd; no voters:
I20260925 10:13:07.198455 11094 raft_consensus.cc:2781] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:07.198658 11094 raft_consensus.cc:686] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 1 LEADER]: Becoming Leader. State: Replica: 65845fb86457409cbf24d3143cbe5494, State: Running, Role: LEADER
I20260925 10:13:07.198798 11094 consensus_queue.cc:237] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } }
I20260925 10:13:07.199676 9817 catalog_manager.cc:5495] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65845fb86457409cbf24d3143cbe5494 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "65845fb86457409cbf24d3143cbe5494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65845fb86457409cbf24d3143cbe5494" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 } health_report { overall_health: UNKNOWN } } }
I20260925 10:13:07.209983 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.207444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.3.152.195:34029)
0925 10:13:07.207511 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.207513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.207523 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:07.207583 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.207584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.207604 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.207695 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:07.207830 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.208399 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.209156 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.209374 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.209452 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.209625 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:07.209752 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:07.209847 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.209873 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.209897 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:13:07.211880 10879 raft_consensus.cc:1261] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 1 FOLLOWER]: Refusing update from remote peer 65845fb86457409cbf24d3143cbe5494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:07.211936 10742 raft_consensus.cc:1261] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 1 FOLLOWER]: Refusing update from remote peer 65845fb86457409cbf24d3143cbe5494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:07.212388 11096 consensus_queue.cc:1035] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f50cb869fa94bc69e439a6a12821286" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:07.212517 11094 consensus_queue.cc:1035] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8884c173f64d429c8e8d1976af0804bd" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:07.215163 11100 mvcc.cc:205] Tried to move back new op lower bound from 7333196542817341440 to 7333196542767058944. Current Snapshot: MvccSnapshot[applied={T|T < 7333196542817341440}]
I20260925 10:13:07.215170 11099 mvcc.cc:205] Tried to move back new op lower bound from 7333196542817341440 to 7333196542767058944. Current Snapshot: MvccSnapshot[applied={T|T < 7333196542817341440}]
I20260925 10:13:07.218416 9817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:07.218528 9817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:07.219986 9817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a1134c071b14f8c8882da1d9e4e9328: Sending DeleteTablet for 3 replicas of tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c
I20260925 10:13:07.220342 10859 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:07 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:33754
I20260925 10:13:07.220396 10722 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:07 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:32790
I20260925 10:13:07.220681 10995 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ca3fdb48b914b2aab1a5a32fe36ca2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:07 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:45138
I20260925 10:13:07.220933 11111 tablet_replica.cc:330] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: stopping tablet replica
I20260925 10:13:07.221076 11111 raft_consensus.cc:2229] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:07.221242 11111 raft_consensus.cc:2258] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:07.221396 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 9786
I20260925 10:13:07.221379 11110 tablet_replica.cc:330] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: stopping tablet replica
I20260925 10:13:07.221525 11110 raft_consensus.cc:2229] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:07.221634 11110 raft_consensus.cc:2258] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:07.221925 11111 ts_tablet_manager.cc:1905] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:07.222090 11110 ts_tablet_manager.cc:1905] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:07.224124 11109 tablet_replica.cc:330] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: stopping tablet replica
I20260925 10:13:07.224284 11109 raft_consensus.cc:2229] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:07.224409 11109 raft_consensus.cc:2258] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:07.224591 11110 ts_tablet_manager.cc:1918] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:07.224674 11110 log.cc:1196] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/2ca3fdb48b914b2aab1a5a32fe36ca2c
I20260925 10:13:07.224963 11110 ts_tablet_manager.cc:1939] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 8884c173f64d429c8e8d1976af0804bd: Deleting consensus metadata
I20260925 10:13:07.227319 11111 ts_tablet_manager.cc:1918] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:07.227475 11111 log.cc:1196] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/2ca3fdb48b914b2aab1a5a32fe36ca2c
I20260925 10:13:07.227852 11111 ts_tablet_manager.cc:1939] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 65845fb86457409cbf24d3143cbe5494: Deleting consensus metadata
W20260925 10:13:07.228848 10950 connection.cc:535] server connection from 127.0.0.1:45138 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 10:13:07.229194 10677 connection.cc:535] server connection from 127.0.0.1:32790 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:13:07.230309 11109 ts_tablet_manager.cc:1905] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:07.232255 11109 ts_tablet_manager.cc:1918] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:07.232342 11109 log.cc:1196] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/2ca3fdb48b914b2aab1a5a32fe36ca2c
I20260925 10:13:07.232638 11109 ts_tablet_manager.cc:1939] T 2ca3fdb48b914b2aab1a5a32fe36ca2c P 3f50cb869fa94bc69e439a6a12821286: Deleting consensus metadata
W20260925 10:13:07.233130 10813 connection.cc:441] server connection from 127.0.0.1:33754 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33754 (request call id 1) could send its response
I20260925 10:13:07.233773 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.220913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.3.152.252:32799)
0925 10:13:07.220955 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.220957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.223450 (+ 2493us) server_negotiation.cc:407] Connection header received
0925 10:13:07.223898 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.223900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.223917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.223949 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.225292 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.226101 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.227222 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.227610 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.227680 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.229565 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.229566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.229566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.229575 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.232125 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.232596 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.232597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.232598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.232658 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.232877 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.232877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.232878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.233026 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.233114 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.233578 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.233677 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:13:07.240620 10792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.231018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.3.152.252:32799)
0925 10:13:07.231063 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.231065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.232901 (+ 1836us) server_negotiation.cc:407] Connection header received
0925 10:13:07.232991 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.232992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.233002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.233031 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.234260 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.235017 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.235899 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.236159 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.236246 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.237700 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.237701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.237702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.237708 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.239471 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.239794 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.239795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.239795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.239853 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.240101 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.240102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.240102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.240216 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.240337 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.240430 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.240509 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:13:07.242525 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:07.234449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.3.152.253:42821)
0925 10:13:07.234497 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:07.234499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:07.234605 (+ 106us) server_negotiation.cc:407] Connection header received
0925 10:13:07.234695 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:07.234696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:07.234710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:07.234755 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:07.235423 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.236317 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.237368 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:07.237690 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:07.237779 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:07.239382 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:07.239383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:07.239384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:07.239391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:07.241328 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.241727 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.241735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.241735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.241794 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:07.241995 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:07.241996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:07.241996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:07.242153 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:07.242258 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:07.242335 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:13:07.242422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:13:08.216707 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:08.216377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:44535)
0925 10:13:08.216514 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:08.216637 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260925 10:13:08.217212 10789 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:08.218699 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:08.218446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:33243)
0925 10:13:08.218582 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:08.218661 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260925 10:13:08.218853 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:08.218606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:33905)
0925 10:13:08.218723 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:08.218807 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260925 10:13:08.218914 11062 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
W20260925 10:13:08.219051 10926 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33181 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:08.721350 11123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:08.721491 11123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:08.721870 11123 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:08.722314 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:08.722102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46904)
0925 10:13:08.722219 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:08.722275 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260925 10:13:08.722477 10400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:08.722915 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:08.722791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:46910)
0925 10:13:08.722860 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:08.722894 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 10:13:08.723018 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:08.723099 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:08.723230 11123 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:09.217885 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:09.217600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:58175)
0925 10:13:09.217750 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:09.217797 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 10:13:09.219435 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:09.219259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:46619)
0925 10:13:09.219355 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:09.219397 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":56,"threads_started":1}
I20260925 10:13:09.219545 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:09.219360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:34483)
0925 10:13:09.219458 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:09.219482 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260925 10:13:10.218843 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:10.218423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:49575)
0925 10:13:10.218723 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:10.218779 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":86,"threads_started":1}
I20260925 10:13:10.220214 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:10.219948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:42507)
0925 10:13:10.220098 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:10.220135 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260925 10:13:10.220206 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:10.219948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:58825)
0925 10:13:10.220097 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:10.220134 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260925 10:13:10.886693 11132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:10.886812 11132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:10.886931 11132 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:10.887373 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:10.887122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:47198)
0925 10:13:10.887274 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:10.887319 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260925 10:13:10.887915 11133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:10.887799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:47206)
0925 10:13:10.887857 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:10.887893 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:13:10.888036 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:10.888110 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:10.888208 11132 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:11.220031 11135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:11.219702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:46497)
0925 10:13:11.219884 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:11.219938 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":130,"threads_started":1}
I20260925 10:13:11.220942 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:11.220700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:59091)
0925 10:13:11.220845 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:11.220887 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260925 10:13:11.220981 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:11.220764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:38481)
0925 10:13:11.220910 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:11.220946 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260925 10:13:12.220736 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:12.220461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:53433)
0925 10:13:12.220648 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:12.220691 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260925 10:13:12.221616 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:12.221422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:42341)
0925 10:13:12.221538 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:12.221580 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260925 10:13:12.221614 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:12.221434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:59451)
0925 10:13:12.221538 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:12.221579 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":78,"threads_started":1}
I20260925 10:13:12.630365 11142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:12.630481 11142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:12.630666 11142 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:12.631057 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:12.630875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58620)
0925 10:13:12.630973 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:12.631020 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":56,"threads_started":1}
I20260925 10:13:12.631469 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:12.631372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58622)
0925 10:13:12.631423 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:12.631448 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:13:12.631551 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:12.631590 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:12.631671 11142 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:13.221428 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:13.221179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:36533)
0925 10:13:13.221322 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:13.221373 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260925 10:13:13.222213 11146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:13.222030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:49003)
0925 10:13:13.222134 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:13.222168 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260925 10:13:13.222368 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:13.222165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:50799)
0925 10:13:13.222266 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:13.222317 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260925 10:13:14.222064 11148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:14.221810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:59109)
0925 10:13:14.221953 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:14.222003 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260925 10:13:14.222929 11149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:14.222667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:45673)
0925 10:13:14.222823 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:14.222874 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":58,"threads_started":1}
I20260925 10:13:14.223078 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:14.222856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:52733)
0925 10:13:14.222967 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:14.223023 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260925 10:13:15.222749 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:15.222482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:34585)
0925 10:13:15.222619 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:15.222691 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 10:13:15.223584 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:15.223401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:57001)
0925 10:13:15.223513 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:15.223548 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260925 10:13:15.223660 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:15.223449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:45751)
0925 10:13:15.223597 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:15.223625 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":45,"threads_started":1}
I20260925 10:13:16.223555 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:16.223246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:59047)
0925 10:13:16.223440 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:16.223492 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":85,"threads_started":1}
I20260925 10:13:16.224313 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:16.224127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:50419)
0925 10:13:16.224237 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:16.224271 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260925 10:13:16.224325 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:16.224133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:33311)
0925 10:13:16.224249 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:16.224290 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260925 10:13:16.227468 11157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:16.227626 11157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:16.227865 11157 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:16.228226 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:16.228058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58892)
0925 10:13:16.228154 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:16.228194 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
W20260925 10:13:16.228317 10400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260925 10:13:16.228820 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:16.228685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58898)
0925 10:13:16.228761 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:16.228796 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 10:13:16.228927 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:16.229007 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:16.229127 11157 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:17.224381 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:17.224127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:42635)
0925 10:13:17.224265 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:17.224317 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260925 10:13:17.224884 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:17.224712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:41567)
0925 10:13:17.224818 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:17.224850 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260925 10:13:17.224912 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:17.224706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:40321)
0925 10:13:17.224820 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:17.224871 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":45,"threads_started":1}
I20260925 10:13:18.225242 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:18.224914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:52807)
0925 10:13:18.225124 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:18.225176 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260925 10:13:18.225472 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:18.225256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:54329)
0925 10:13:18.225399 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:18.225434 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260925 10:13:18.225623 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:18.225384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:50467)
0925 10:13:18.225506 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:18.225568 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":50,"threads_started":1}
I20260925 10:13:19.226030 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:19.225817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:59483)
0925 10:13:19.225945 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:19.225975 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":44,"threads_started":1}
I20260925 10:13:19.226065 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:19.225764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:50373)
0925 10:13:19.225935 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:19.225976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260925 10:13:19.226204 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:19.225968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:47637)
0925 10:13:19.226087 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:19.226142 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260925 10:13:20.183754 11169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:20.183867 11169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:20.184082 11169 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:20.184489 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:20.184289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:59154)
0925 10:13:20.184403 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:20.184449 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260925 10:13:20.184906 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:20.184805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:59166)
0925 10:13:20.184857 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:20.184886 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:13:20.184994 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:20.185078 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:20.185207 11169 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:20.226842 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:20.226538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:35919)
0925 10:13:20.226737 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:20.226781 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":65,"threads_started":1}
I20260925 10:13:20.226850 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:20.226589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:44107)
0925 10:13:20.226709 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:20.226753 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260925 10:13:20.226929 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:20.226673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:34565)
0925 10:13:20.226849 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:20.226883 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":48,"threads_started":1}
I20260925 10:13:21.227607 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:21.227358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:36433)
0925 10:13:21.227501 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:21.227547 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260925 10:13:21.227703 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:21.227471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:44867)
0925 10:13:21.227613 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:21.227652 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":47,"threads_started":1}
I20260925 10:13:21.227736 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:21.227379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:60071)
0925 10:13:21.227662 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:21.227693 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":60,"threads_started":1}
I20260925 10:13:22.228410 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:22.228174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:52193)
0925 10:13:22.228313 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:22.228354 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260925 10:13:22.228600 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:22.228207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:56571)
0925 10:13:22.228520 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:22.228556 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":52,"threads_started":1}
I20260925 10:13:22.228616 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:22.228174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:54657)
0925 10:13:22.228540 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:22.228578 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":64,"threads_started":1}
I20260925 10:13:23.229269 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:23.229018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:51925)
0925 10:13:23.229158 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:23.229203 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 10:13:23.229360 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:23.229026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:46443)
0925 10:13:23.229222 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:23.229263 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260925 10:13:23.229718 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:23.229314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:38163)
0925 10:13:23.229601 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:23.229676 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":41,"threads_started":1}
I20260925 10:13:23.997077 11184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:23.997202 11184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:23.997419 11184 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:23.997749 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:23.997543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58644)
0925 10:13:23.997662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:23.997712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
W20260925 10:13:23.997838 10400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 10:13:23.998217 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:23.998099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:58656)
0925 10:13:23.998155 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:23.998189 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:13:23.998297 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:23.998333 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:23.998399 11184 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:24.230259 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:24.229927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:38121)
0925 10:13:24.230153 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:24.230197 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":63,"threads_started":1}
I20260925 10:13:24.230350 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:24.230176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:45349)
0925 10:13:24.230282 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:24.230309 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":45,"threads_started":1}
I20260925 10:13:24.230376 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:24.230113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:50835)
0925 10:13:24.230284 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:24.230325 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":124,"threads_started":1}
I20260925 10:13:25.230983 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:25.230719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:40831)
0925 10:13:25.230861 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:25.230913 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260925 10:13:25.231247 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:25.230839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:34887)
0925 10:13:25.231136 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:25.231174 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":66,"threads_started":1}
I20260925 10:13:25.231320 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:25.230980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:38273)
0925 10:13:25.231246 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:25.231280 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":56,"threads_started":1}
I20260925 10:13:26.231873 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:26.231596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:35239)
0925 10:13:26.231767 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:26.231810 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260925 10:13:26.231916 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:26.231685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:56063)
0925 10:13:26.231842 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:26.231874 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260925 10:13:26.232129 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:26.231888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:47285)
0925 10:13:26.232052 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:26.232087 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":54,"threads_started":1}
I20260925 10:13:27.232715 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:27.232440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:36597)
0925 10:13:27.232608 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:27.232652 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260925 10:13:27.232726 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:27.232479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:55771)
0925 10:13:27.232643 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:27.232679 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":55,"threads_started":1}
I20260925 10:13:27.232980 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:27.232538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:33375)
0925 10:13:27.232899 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:27.232936 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":58,"threads_started":1}
I20260925 10:13:28.233614 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:28.233350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:37291)
0925 10:13:28.233523 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:28.233562 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
I20260925 10:13:28.233614 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:28.233350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:39119)
0925 10:13:28.233520 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:28.233563 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260925 10:13:28.233738 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:28.233331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:54201)
0925 10:13:28.233646 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:28.233687 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":67,"threads_started":1}
I20260925 10:13:29.234277 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:29.234010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:53649)
0925 10:13:29.234156 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:29.234210 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
I20260925 10:13:29.234572 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:29.234285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:52791)
0925 10:13:29.234489 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:29.234524 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":52,"threads_started":1}
I20260925 10:13:29.234687 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:29.234413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:59457)
0925 10:13:29.234598 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:29.234636 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":47,"threads_started":1}
I20260925 10:13:30.235289 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:30.234878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:39011)
0925 10:13:30.235182 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:30.235231 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":64,"threads_started":1}
I20260925 10:13:30.235519 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:30.235152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:47679)
0925 10:13:30.235441 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:30.235477 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":56,"threads_started":1}
I20260925 10:13:30.235934 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:30.235660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:46063)
0925 10:13:30.235839 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:30.235896 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":57,"threads_started":1}
I20260925 10:13:31.236166 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:31.235792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:40381)
0925 10:13:31.236058 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:31.236107 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":69,"threads_started":1}
I20260925 10:13:31.236542 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:31.236168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:53529)
0925 10:13:31.236451 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:31.236491 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":68,"threads_started":1}
I20260925 10:13:31.236574 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:31.236388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:48757)
0925 10:13:31.236493 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:31.236524 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":43,"threads_started":1}
I20260925 10:13:31.739746 11211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a1134c071b14f8c8882da1d9e4e9328)
I20260925 10:13:31.739854 11211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a1134c071b14f8c8882da1d9e4e9328" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33181 } } peers { permanent_uuid: "108023a6f92c4f74a2a889b0749a4de3" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 42821 } attrs { promote: false } } peers { permanent_uuid: "e9f581581e2b4c009676a895d81c973c" member_type: NON_VOTER last_known_addr { host: "127.3.152.252" port: 32799 } attrs { promote: true } }
I20260925 10:13:31.740082 11211 leader_election.cc:288] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181)
I20260925 10:13:31.740475 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:31.740261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:59026)
0925 10:13:31.740384 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:31.740434 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
W20260925 10:13:31.740629 10400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:31.740985 11212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:31.740881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.0.0.1:59040)
0925 10:13:31.740936 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:31.740962 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:13:31.741070 10400 leader_election.cc:334] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a1134c071b14f8c8882da1d9e4e9328 (127.3.152.254:33181): Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
I20260925 10:13:31.741114 10400 leader_election.cc:302] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 108023a6f92c4f74a2a889b0749a4de3; no voters: 2a1134c071b14f8c8882da1d9e4e9328
I20260925 10:13:31.741181 11211 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 108023a6f92c4f74a2a889b0749a4de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:13:32.237130 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:32.236835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:60199)
0925 10:13:32.237024 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:32.237064 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
I20260925 10:13:32.237340 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:32.236949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:57823)
0925 10:13:32.237226 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:32.237275 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":44,"threads_started":1}
I20260925 10:13:32.237673 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:32.237021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:47735)
0925 10:13:32.237577 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:32.237618 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":58,"threads_started":1}
I20260925 10:13:33.238148 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:33.237737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:58657)
0925 10:13:33.238030 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:33.238075 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":85,"threads_started":1}
I20260925 10:13:33.238343 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:33.238140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:46971)
0925 10:13:33.238261 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:33.238295 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260925 10:13:33.238349 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:33.237895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:46767)
0925 10:13:33.238281 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:33.238311 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":58,"threads_started":1}
I20260925 10:13:34.239025 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:34.238791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:59775)
0925 10:13:34.238925 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:34.238966 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 10:13:34.239310 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:34.238918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:39667)
0925 10:13:34.239225 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:34.239263 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":44,"threads_started":1}
I20260925 10:13:34.239399 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:34.238825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:57809)
0925 10:13:34.239317 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:34.239352 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":57,"threads_started":1}
I20260925 10:13:35.240164 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:35.239636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:59431)
0925 10:13:35.240060 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:35.240107 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260925 10:13:35.240198 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:35.239913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:37137)
0925 10:13:35.240087 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:35.240142 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260925 10:13:35.240506 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:35.239832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:39969)
0925 10:13:35.240411 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:35.240455 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":50,"threads_started":1}
I20260925 10:13:36.241343 11227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:36.240723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:49349)
0925 10:13:36.241252 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:36.241293 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":86,"threads_started":1}
I20260925 10:13:36.241339 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:36.241010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:52673)
0925 10:13:36.241240 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:36.241279 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":51,"threads_started":1}
I20260925 10:13:36.241400 11226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:36.240689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:40829)
0925 10:13:36.241327 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:36.241360 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":112,"threads_started":1}
I20260925 10:13:37.242273 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:37.241843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.194:55319)
0925 10:13:37.242166 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:37.242212 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":82,"threads_started":1}
I20260925 10:13:37.242479 11231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:37.242085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.195:56793)
0925 10:13:37.242337 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:37.242376 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":68,"threads_started":1}
I20260925 10:13:37.242532 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:37.242085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33181 (local address 127.3.152.193:43303)
0925 10:13:37.242450 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:37.242489 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:33181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":59,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.3.152.254:33181,127.3.152.253:42821,127.3.152.252:32799, num_attempts: 1)
I20260925 10:13:37.458806 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10563
I20260925 10:13:37.464073 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10796
I20260925 10:13:37.468326 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10932
I20260925 10:13:37.473380 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10283
I20260925 10:13:37.478116 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 10470
2026-09-25T10:13:37Z chronyd exiting
I20260925 10:13:37.494349 3683 test_util.cc:175] -----------------------------------------------
I20260925 10:13:37.494423 3683 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790331159020724-3683-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33167 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:13:37 dist-test-slave-dzwq krb5kdc[11234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:13:37 dist-test-slave-dzwq krb5kdc[11234](info): set up 2 sockets
Sep 25 10:13:37 dist-test-slave-dzwq krb5kdc[11234](info): commencing operation
krb5kdc: starting...
W20260925 10:13:38.533816 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:13:38 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab.
Entry for principal oryx/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab.
I20260925 10:13:38.657322 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41947
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab
--principal=oryx/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:38.741377 11247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:38.741600 11247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:38.741626 11247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:38.742834 11247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:38.742882 11247 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 10:13:38.742895 11247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:38.742909 11247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:38.742921 11247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:38.742933 11247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:38.744490 11247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41947
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.254.keytab
--principal=oryx/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:38.744692 11247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:38.744874 11247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:38.750207 11255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:38.750219 11253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:38.750315 11247 server_base.cc:1034] running on GCE node
W20260925 10:13:38.750223 11252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:38.750779 11247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:38.751094 11247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:38.752238 11247 hybrid_clock.cc:648] HybridClock initialized: now 1790331218752214 us; error 38 us; skew 500 ppm
Sep 25 10:13:38 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:38.754807 11247 init.cc:377] Logged in from keytab as oryx/127.3.152.254@KRBTEST.COM (short username oryx)
I20260925 10:13:38.755300 11247 webserver.cc:460] Webserver started at http://127.3.152.254:43765/ using document root <none> and password file <none>
I20260925 10:13:38.755540 11247 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:38.755605 11247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:38.755755 11247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:38.756916 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "d641a34d07214e839df9740913b44bcd"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "fa28d824e06a762c3af668fd58a20489"
server_key_iv: "bf3897ba6e757a8dd0d6f7454af073b3"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.757306 11247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "d641a34d07214e839df9740913b44bcd"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "fa28d824e06a762c3af668fd58a20489"
server_key_iv: "bf3897ba6e757a8dd0d6f7454af073b3"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.758641 11247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:38.759344 11262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:38.759507 11247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:38.759608 11247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "d641a34d07214e839df9740913b44bcd"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "fa28d824e06a762c3af668fd58a20489"
server_key_iv: "bf3897ba6e757a8dd0d6f7454af073b3"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.759680 11247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:38.777782 11247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:38.780311 11247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:38.780481 11247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:38.785508 11247 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41947
I20260925 10:13:38.785552 11324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41947 every 8 connection(s)
I20260925 10:13:38.785912 11247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:38.786602 11325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:38.789266 11325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Bootstrap starting.
I20260925 10:13:38.789920 11325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:38.790238 11325 log.cc:824] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:38.791057 11325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: No bootstrap required, opened a new log
I20260925 10:13:38.792695 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 11247
I20260925 10:13:38.792780 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:38.792972 3683 external_mini_cluster.cc:1428] Setting key d002f20eca405c0610dc42d772882ea3
I20260925 10:13:38.793452 11325 raft_consensus.cc:348] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } }
I20260925 10:13:38.793777 11325 raft_consensus.cc:374] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:38.793844 11325 raft_consensus.cc:729] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d641a34d07214e839df9740913b44bcd, State: Initialized, Role: FOLLOWER
I20260925 10:13:38.794000 11325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } }
I20260925 10:13:38.794097 11325 raft_consensus.cc:388] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:38.794153 11325 raft_consensus.cc:482] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:38.794227 11325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:38.794807 11325 raft_consensus.cc:504] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } }
I20260925 10:13:38.794926 11325 leader_election.cc:302] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d641a34d07214e839df9740913b44bcd; no voters:
I20260925 10:13:38.795085 11325 leader_election.cc:288] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:38.795115 11330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:38.795248 11330 raft_consensus.cc:686] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [term 1 LEADER]: Becoming Leader. State: Replica: d641a34d07214e839df9740913b44bcd, State: Running, Role: LEADER
I20260925 10:13:38.795373 11325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:38.795394 11330 consensus_queue.cc:237] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } }
I20260925 10:13:38.795815 11330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d641a34d07214e839df9740913b44bcd. Latest consensus state: current_term: 1 leader_uuid: "d641a34d07214e839df9740913b44bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } } }
I20260925 10:13:38.795825 11331 sys_catalog.cc:455] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d641a34d07214e839df9740913b44bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641a34d07214e839df9740913b44bcd" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41947 } } }
I20260925 10:13:38.796060 11331 sys_catalog.cc:458] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:38.796288 11330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:38.796672 11339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:38.797369 11339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:38.798954 11339 catalog_manager.cc:1347] Generated new cluster ID: e9654810637046189f797bfd8ebfc568
I20260925 10:13:38.799013 11339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:13:38 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:38.805045 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:38.793537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.3.152.254:41947)
0925 10:13:38.793728 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:38.793739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:38.793777 (+ 38us) server_negotiation.cc:407] Connection header received
0925 10:13:38.793916 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:38.793924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:38.793985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:38.794131 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:38.794861 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.796247 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.797370 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.797841 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.797959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:38.800829 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:38.800834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:38.800834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:38.800846 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:38.803313 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.803740 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.803742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.803743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.803853 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.804105 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.804106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.804106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.804412 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:38.804571 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:38.804692 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 10:13:38.804827 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260925 10:13:38.811151 11339 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:38.811901 11339 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:38.817333 11339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Generated new TSK 0
I20260925 10:13:38.817538 11339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
I20260925 10:13:38.846776 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41947
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:38.929759 11351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:38.929980 11351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:38.930006 11351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:38.931173 11351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:38.931217 11351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:38.931238 11351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:13:38.931298 11351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:38.932885 11351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:38.933153 11351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:38.933391 11351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:38.938443 11359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:38.938443 11356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:38.938571 11357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:38.938781 11351 server_base.cc:1034] running on GCE node
I20260925 10:13:38.939011 11351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:38.939260 11351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:38.940397 11351 hybrid_clock.cc:648] HybridClock initialized: now 1790331218940385 us; error 26 us; skew 500 ppm
Sep 25 10:13:38 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:38.943040 11351 init.cc:377] Logged in from keytab as oryx/127.3.152.193@KRBTEST.COM (short username oryx)
I20260925 10:13:38.943493 11351 webserver.cc:460] Webserver started at http://127.3.152.193:46639/ using document root <none> and password file <none>
I20260925 10:13:38.943715 11351 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:38.943760 11351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:38.943852 11351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:38.944929 11351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "db37352c199c4cdda2bfe23e5fed4e0b"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "a90665aacb4b848a62f13a99c6d73141"
server_key_iv: "3792b68dadcd48861ca773c04ce364ad"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.945250 11351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "db37352c199c4cdda2bfe23e5fed4e0b"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "a90665aacb4b848a62f13a99c6d73141"
server_key_iv: "3792b68dadcd48861ca773c04ce364ad"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.946550 11351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:38.947244 11366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:38.947412 11351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:38.947515 11351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "db37352c199c4cdda2bfe23e5fed4e0b"
format_stamp: "Formatted at 2026-09-25 10:13:38 on dist-test-slave-dzwq"
server_key: "a90665aacb4b848a62f13a99c6d73141"
server_key_iv: "3792b68dadcd48861ca773c04ce364ad"
server_key_version: "encryptionkey@0"
I20260925 10:13:38.947589 11351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:38.967034 11351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:38.969583 11351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:38.969751 11351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:38.970367 11351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:38.970423 11351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:38.970469 11351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:38.970504 11351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:13:38 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:38.977885 11351 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:37453
I20260925 10:13:38.977972 11481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:37453 every 8 connection(s)
I20260925 10:13:38.978289 11351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:38.980257 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:38.970816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:38345 (local address 127.3.152.254:41947)
0925 10:13:38.970885 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:38.970888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:38.971049 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:13:38.971540 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:38.971541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:38.971559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:38.971656 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:38.972390 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.973243 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.974368 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.974766 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.974856 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:38.977345 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:38.977346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:38.977347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:38.977355 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:38.979024 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.979387 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.979388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.979389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.979440 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.979642 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.979643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.979643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.979748 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:38.979799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:38.980114 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:13:38.980186 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:13:38.980427 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:38.970754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.193:38345)
0925 10:13:38.970917 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:38.970958 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:38.971446 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:38.971702 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:38.971705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:38.972022 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:38.972296 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:38.972301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.973356 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.973357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:38.974266 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:38.974269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.974950 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.974951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:38.975129 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:38.975717 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:38.975732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:38.977232 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:38.979094 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:38.979099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:38.979100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:38.979320 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:38.979494 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:38.979495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:38.979495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:38.979591 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:38.979802 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:38.979807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:38.979945 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 10:13:38.980094 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 10:13:38.980224 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260925 10:13:38.980916 11482 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41947
I20260925 10:13:38.980997 11482 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:38.981202 11482 heartbeater.cc:511] Master 127.3.152.254:41947 requested a full tablet report, sending...
I20260925 10:13:38.981761 11278 ts_manager.cc:194] Registered new tserver with Master: db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453)
I20260925 10:13:38.982293 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 11351
I20260925 10:13:38.982400 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:38.982582 3683 external_mini_cluster.cc:1428] Setting key 832c4f80e161aea048db10b3ecfd1b6b
I20260925 10:13:38.982722 11278 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:38345
I20260925 10:13:38.987804 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:38.981973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.193:57017)
0925 10:13:38.982160 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:38.982169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:38.982235 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:38.982356 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:38.982357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:38.982536 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:38.982591 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:38.982593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.983527 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.983528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:38.984116 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:38.984119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.984494 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.984495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:38.984572 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:38.985039 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:38.985043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:38.985321 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:38.986807 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:38.986808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:38.986808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:38.986982 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:38.987202 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:38.987202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:38.987203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:38.987261 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:38.987537 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:38.987538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:38.987583 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:13:38.987648 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:13:38.987733 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260925 10:13:38.987938 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:38.982033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:57017 (local address 127.3.152.254:41947)
0925 10:13:38.982084 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:38.982086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:38.982221 (+ 135us) server_negotiation.cc:407] Connection header received
0925 10:13:38.982294 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:38.982296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:38.982306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:38.982355 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:38.982811 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.983431 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.984209 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:38.984411 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:38.984492 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:38.985408 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:38.985409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:38.985410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:38.985417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:38.986715 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.987063 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.987064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.987065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.987122 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:38.987329 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:38.987330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:38.987331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:38.987474 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:38.987545 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:38.987696 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 10:13:38.987799 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for oryx/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
I20260925 10:13:39.011824 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41947
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:39.095806 11491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:39.096087 11491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:39.096130 11491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:39.097494 11491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:39.097568 11491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:39.097605 11491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:13:39.097641 11491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:39.099272 11491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:39.099561 11491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:39.099812 11491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:39.105180 11496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:39.105172 11497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:39.105234 11491 server_base.cc:1034] running on GCE node
W20260925 10:13:39.105172 11499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:39.105595 11491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:39.105813 11491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:39.106943 11491 hybrid_clock.cc:648] HybridClock initialized: now 1790331219106913 us; error 47 us; skew 500 ppm
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:39.109660 11491 init.cc:377] Logged in from keytab as oryx/127.3.152.194@KRBTEST.COM (short username oryx)
I20260925 10:13:39.110102 11491 webserver.cc:460] Webserver started at http://127.3.152.194:43633/ using document root <none> and password file <none>
I20260925 10:13:39.110317 11491 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:39.110359 11491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:39.110453 11491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:39.111511 11491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "33d0c68561ed4ee39349f604a1135cbc"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "fc566d71619e5d88efa141435f762da5"
server_key_iv: "01b5270e7d2c282954ae5c91fc363951"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.111830 11491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "33d0c68561ed4ee39349f604a1135cbc"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "fc566d71619e5d88efa141435f762da5"
server_key_iv: "01b5270e7d2c282954ae5c91fc363951"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.113315 11491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:39.114161 11506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.114357 11491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:39.114467 11491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "33d0c68561ed4ee39349f604a1135cbc"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "fc566d71619e5d88efa141435f762da5"
server_key_iv: "01b5270e7d2c282954ae5c91fc363951"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.114543 11491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:39.124399 11491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:39.126916 11491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:39.127099 11491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:39.127632 11491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:39.127692 11491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.127737 11491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:39.127808 11491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:39.135136 11491 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:45905
I20260925 10:13:39.135252 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:45905 every 8 connection(s)
I20260925 10:13:39.135556 11491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:39.137187 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 11491
I20260925 10:13:39.137343 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:39.137566 3683 external_mini_cluster.cc:1428] Setting key d67c475b4bb477a2c58b6b69755c078f
I20260925 10:13:39.138701 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.128312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42593 (local address 127.3.152.254:41947)
0925 10:13:39.128382 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.128385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.128580 (+ 195us) server_negotiation.cc:407] Connection header received
0925 10:13:39.129110 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.129112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.129126 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.129225 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.130134 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.130985 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.132467 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.132686 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.132775 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.135452 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.135453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.135454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.135463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.137071 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.137621 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.137622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.137623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.137702 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.137947 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.137948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.137948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.138098 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.138181 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.138551 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.138635 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:13:39.139019 11515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.128278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.194:42593)
0925 10:13:39.128439 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.128481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.129002 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.129285 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.129288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.129673 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:39.129999 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.130007 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.131119 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.131121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.132350 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.132353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.132802 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.132803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.132955 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.133560 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:39.133569 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:39.135347 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:39.137167 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.137175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.137178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.137520 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.137793 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.137793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.137794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.137898 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.138183 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:39.138189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:39.138369 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.138498 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.138689 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 10:13:39.140223 11622 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41947
I20260925 10:13:39.140340 11622 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:39.140529 11622 heartbeater.cc:511] Master 127.3.152.254:41947 requested a full tablet report, sending...
I20260925 10:13:39.141371 11279 ts_manager.cc:194] Registered new tserver with Master: 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905)
I20260925 10:13:39.141916 11279 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:42593
WARNING: no policy specified for oryx/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:13:39.148499 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.141480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53085 (local address 127.3.152.254:41947)
0925 10:13:39.141524 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.141526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.141635 (+ 109us) server_negotiation.cc:407] Connection header received
0925 10:13:39.141699 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.141700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.141711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.141768 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.142110 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.142921 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.144165 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.144351 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.144411 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.145631 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.145633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.145633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.145640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.147231 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.147682 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.147683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.147683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.147741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.147955 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.147956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.147957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.148149 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.148210 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.148367 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.148433 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:39.148509 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.141412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.194:53085)
0925 10:13:39.141564 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.141575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.141635 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.141771 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.141772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.141981 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:39.142050 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.142052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.143025 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.143026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.144083 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.144086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.144431 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.144432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.144527 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.145058 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:39.145063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:39.145470 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:39.147323 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.147324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.147325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.147592 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.147817 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.147818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.147818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.147888 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.148231 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:39.148232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:39.148276 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.148345 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.148429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
I20260925 10:13:39.167898 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41947
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:39.251405 11631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:39.251700 11631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:39.251732 11631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:39.253017 11631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:39.253067 11631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:39.253088 11631 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 10:13:39.253111 11631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:39.254680 11631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.152.254:41947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:39.254895 11631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:39.255088 11631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:39.260299 11639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:39.260300 11636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:39.260299 11637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:39.260392 11631 server_base.cc:1034] running on GCE node
I20260925 10:13:39.260866 11631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:39.261106 11631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:39.262251 11631 hybrid_clock.cc:648] HybridClock initialized: now 1790331219262236 us; error 32 us; skew 500 ppm
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:39.265014 11631 init.cc:377] Logged in from keytab as oryx/127.3.152.195@KRBTEST.COM (short username oryx)
I20260925 10:13:39.265484 11631 webserver.cc:460] Webserver started at http://127.3.152.195:41883/ using document root <none> and password file <none>
I20260925 10:13:39.265751 11631 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:39.265802 11631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:39.265974 11631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:39.267143 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "4edcef40642a487e9521625c334b8bc8"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "cb4ffdcf7cc1683e336a99cd2aa2a95d"
server_key_iv: "ccbf55aa50cfe1715ad6813ee45dddc2"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.267535 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "4edcef40642a487e9521625c334b8bc8"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "cb4ffdcf7cc1683e336a99cd2aa2a95d"
server_key_iv: "ccbf55aa50cfe1715ad6813ee45dddc2"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.269054 11631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:13:39.269840 11646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.270008 11631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:39.270109 11631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "4edcef40642a487e9521625c334b8bc8"
format_stamp: "Formatted at 2026-09-25 10:13:39 on dist-test-slave-dzwq"
server_key: "cb4ffdcf7cc1683e336a99cd2aa2a95d"
server_key_iv: "ccbf55aa50cfe1715ad6813ee45dddc2"
server_key_version: "encryptionkey@0"
I20260925 10:13:39.270186 11631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:39.298775 11631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:39.301385 11631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:39.301560 11631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:39.302102 11631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:39.302165 11631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.302219 11631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:39.302258 11631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.309617 11631 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:46629
I20260925 10:13:39.309772 11761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:46629 every 8 connection(s)
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:39.310050 11631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:39.313450 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.302802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:54215 (local address 127.3.152.254:41947)
0925 10:13:39.302859 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.302862 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.303017 (+ 155us) server_negotiation.cc:407] Connection header received
0925 10:13:39.303770 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.303771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.303787 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.303854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.304903 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.305664 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.307463 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.307683 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.307745 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.310618 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.310620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.310620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.310628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.311884 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.312350 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.312351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.312352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.312410 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.312802 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.312803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.312804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.312929 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.312990 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.313305 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.313384 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:13:39.313696 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.302672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.195:54215)
0925 10:13:39.302876 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.302923 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.303605 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.304001 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.304007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.304459 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:39.304786 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.304795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.305776 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.305778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.307352 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.307356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.307776 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.307777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.307986 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.308698 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:39.308712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:39.310523 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:39.312012 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.312020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.312022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.312278 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.312471 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.312472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.312472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.312580 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.313011 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:39.313017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:39.313155 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.313253 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.313391 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260925 10:13:39.313789 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 11631
I20260925 10:13:39.313887 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:39.314065 3683 external_mini_cluster.cc:1428] Setting key e165d7e556eb42141940b3e700888377
I20260925 10:13:39.314301 11762 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41947
I20260925 10:13:39.314392 11762 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:39.314630 11762 heartbeater.cc:511] Master 127.3.152.254:41947 requested a full tablet report, sending...
I20260925 10:13:39.314980 11278 ts_manager.cc:194] Registered new tserver with Master: 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:39.315369 11278 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:54215
I20260925 10:13:39.321239 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.315488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:41687 (local address 127.3.152.254:41947)
0925 10:13:39.315529 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.315531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.315770 (+ 239us) server_negotiation.cc:407] Connection header received
0925 10:13:39.315828 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.315828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.315838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.315886 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.316288 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.316750 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.317562 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.317748 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.317804 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.318833 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.318834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.318835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.318840 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.320267 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.320616 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.320617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.320617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.320669 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.320836 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.320837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.320837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.320941 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.320990 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.321100 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.321160 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:13:39.321250 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.315550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.3.152.195:41687)
0925 10:13:39.315698 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.315710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.315768 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.315890 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.315891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.316142 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:39.316204 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.316207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.316835 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.316835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.317494 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.317496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.317875 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.317876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.317958 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.318443 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:39.318449 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:39.318757 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:39.320366 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.320367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.320367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.320545 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.320727 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.320728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.320728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.320785 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.320991 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:39.320992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:39.321035 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.321085 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.321179 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260925 10:13:39.325861 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:13:39.347469 11329 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:39.341815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.3.152.254:41947)
0925 10:13:39.341880 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.341882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.341892 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:39.341926 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.341927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.341937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.341990 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.342547 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.343054 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.344005 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.344215 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.344272 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.347375 (+ 3103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45410: BlockingRecv error: recv got EOF from 127.0.0.1:45410 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331219, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:39.355690 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.348340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.3.152.254:41947)
0925 10:13:39.348411 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.348413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.348475 (+ 62us) server_negotiation.cc:407] Connection header received
0925 10:13:39.348553 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.348554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.348564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.348613 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.349111 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.349592 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.350366 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.350560 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.350645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.353199 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.353201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.353201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.353208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.354700 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.355048 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.355049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.355049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.355101 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.355263 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.355264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.355264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.355363 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.355434 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.355528 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.355589 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:13:39.357436 11278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:39.359079 11278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:39.365908 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.0.0.1:40664)
0925 10:13:39.361859 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.361869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.361918 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.362231 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.362234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.362237 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.362328 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.362331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363402 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.363403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.364870 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.364873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365582 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.365665 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365680 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.365750 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.365811 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":41,"thread_start_us":112,"threads_started":1}
I20260925 10:13:39.366007 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:45905 (local address 127.0.0.1:39066)
0925 10:13:39.361853 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.361864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.361923 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.362261 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.362262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.362265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.362325 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.362327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363696 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.363697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.364871 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.364873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365738 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.365804 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365807 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.365858 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.365890 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":34,"thread_start_us":81,"threads_started":1}
I20260925 10:13:39.366132 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58464 (local address 127.3.152.195:46629)
0925 10:13:39.362351 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.362356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.362365 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:39.362404 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.362406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.362419 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.362497 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.362590 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363689 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.364824 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365554 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365628 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365721 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.365966 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.366009 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":81,"threads_started":1}
I20260925 10:13:39.366173 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46629 (local address 127.0.0.1:58464)
0925 10:13:39.361786 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.361833 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.361918 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.362491 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.362492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.362494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.362527 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.362528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363767 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.363768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.364723 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.364726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365760 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.365829 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365833 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.365886 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.365926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260925 10:13:39.366348 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.3.152.193:37453)
0925 10:13:39.362059 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.362065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.362075 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:39.362120 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.362122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.362139 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.362229 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.362411 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363324 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.364975 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365477 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365578 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365664 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.366126 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.366172 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260925 10:13:39.366762 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.361670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.3.152.194:45905)
0925 10:13:39.362060 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.362065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.362101 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:13:39.362137 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.362138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.362166 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.362403 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.362466 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.363609 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.364957 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.365659 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.365722 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.365786 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.365881 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.365921 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":90,"threads_started":1}
I20260925 10:13:39.366950 11696 tablet_service.cc:1467] Processing CreateTablet for tablet 230590e18a9440b8b52009cad62f5c21 (DEFAULT_TABLE table=test-table [id=07587c79d80c446785c8b588e7ded290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.366993 11416 tablet_service.cc:1467] Processing CreateTablet for tablet 230590e18a9440b8b52009cad62f5c21 (DEFAULT_TABLE table=test-table [id=07587c79d80c446785c8b588e7ded290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.367344 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230590e18a9440b8b52009cad62f5c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.367358 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230590e18a9440b8b52009cad62f5c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.367344 11556 tablet_service.cc:1467] Processing CreateTablet for tablet 230590e18a9440b8b52009cad62f5c21 (DEFAULT_TABLE table=test-table [id=07587c79d80c446785c8b588e7ded290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.367656 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 230590e18a9440b8b52009cad62f5c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.370725 11790 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Bootstrap starting.
I20260925 10:13:39.371466 11791 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Bootstrap starting.
I20260925 10:13:39.371717 11792 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Bootstrap starting.
I20260925 10:13:39.371896 11790 tablet_bootstrap.cc:654] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.372117 11791 tablet_bootstrap.cc:654] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.372275 11790 log.cc:824] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:39.372397 11792 tablet_bootstrap.cc:654] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.372475 11791 log.cc:824] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:39.372743 11792 log.cc:824] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:39.373155 11790 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: No bootstrap required, opened a new log
I20260925 10:13:39.373242 11790 ts_tablet_manager.cc:1397] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:13:39.373435 11792 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: No bootstrap required, opened a new log
I20260925 10:13:39.373492 11792 ts_tablet_manager.cc:1397] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:39.375070 11790 raft_consensus.cc:348] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.375084 11792 raft_consensus.cc:348] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.375213 11790 raft_consensus.cc:374] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.375237 11790 raft_consensus.cc:729] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.375232 11792 raft_consensus.cc:374] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.375274 11792 raft_consensus.cc:729] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.375365 11790 consensus_queue.cc:260] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.375423 11792 consensus_queue.cc:260] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.375662 11790 ts_tablet_manager.cc:1428] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:13:39.375710 11792 ts_tablet_manager.cc:1428] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:13:39.375763 11622 heartbeater.cc:503] Master 127.3.152.254:41947 was elected leader, sending a full tablet report...
I20260925 10:13:39.375821 11482 heartbeater.cc:503] Master 127.3.152.254:41947 was elected leader, sending a full tablet report...
I20260925 10:13:39.376565 11791 tablet_bootstrap.cc:492] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: No bootstrap required, opened a new log
I20260925 10:13:39.376670 11791 ts_tablet_manager.cc:1397] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260925 10:13:39.378098 11791 raft_consensus.cc:348] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.378244 11791 raft_consensus.cc:374] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.378268 11791 raft_consensus.cc:729] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.378414 11791 consensus_queue.cc:260] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.378690 11791 ts_tablet_manager.cc:1428] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:13:39.378785 11762 heartbeater.cc:503] Master 127.3.152.254:41947 was elected leader, sending a full tablet report...
W20260925 10:13:39.386166 11623 tablet.cc:2369] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:13:39.479175 11483 tablet.cc:2369] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:13:39.560753 11763 tablet.cc:2369] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:13:39.622654 11796 raft_consensus.cc:482] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:39.622799 11796 raft_consensus.cc:504] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.623225 11796 leader_election.cc:288] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629), 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905)
I20260925 10:13:39.626427 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.623530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:41943 (local address 127.3.152.194:45905)
0925 10:13:39.623578 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.623580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.623695 (+ 115us) server_negotiation.cc:407] Connection header received
0925 10:13:39.623749 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.623751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.623768 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.623854 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.624054 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.624610 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.625670 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.626059 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.626121 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.626129 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.626299 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.626348 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:39.626462 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.623518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:37335 (local address 127.3.152.195:46629)
0925 10:13:39.623570 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.623572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.623848 (+ 276us) server_negotiation.cc:407] Connection header received
0925 10:13:39.623885 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.623886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.623895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.623978 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.624167 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.624614 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.625720 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.626078 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.626167 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.626175 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.626357 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.626398 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:13:39.626452 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.623424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:45905 (local address 127.3.152.193:41943)
0925 10:13:39.623591 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.623603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.623668 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.623865 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.623866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.623869 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.623934 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.623937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.624707 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.624708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.625578 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.625580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.626150 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.626151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.626229 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.626233 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.626277 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.626326 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":50,"thread_start_us":74,"threads_started":1}
I20260925 10:13:39.626482 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.623445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46629 (local address 127.3.152.193:37335)
0925 10:13:39.623731 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.623751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.623831 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.624015 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.624016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.624018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.624090 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.624092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.624707 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.624708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.625589 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.625591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.626190 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.626191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.626253 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.626257 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.626337 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.626378 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":17,"thread_start_us":60,"threads_started":1}
I20260925 10:13:39.626997 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230590e18a9440b8b52009cad62f5c21" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc" is_pre_election: true
I20260925 10:13:39.627027 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230590e18a9440b8b52009cad62f5c21" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8" is_pre_election: true
I20260925 10:13:39.627173 11576 raft_consensus.cc:2454] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:39.627195 11716 raft_consensus.cc:2454] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:39.627454 11370 leader_election.cc:302] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edcef40642a487e9521625c334b8bc8, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.627601 11796 raft_consensus.cc:2781] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:39.627663 11796 raft_consensus.cc:482] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:39.627701 11796 raft_consensus.cc:3037] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.628455 11796 raft_consensus.cc:504] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.628620 11796 leader_election.cc:288] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Requested vote from peers 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629), 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905)
I20260925 10:13:39.628798 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230590e18a9440b8b52009cad62f5c21" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8"
I20260925 10:13:39.628798 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "230590e18a9440b8b52009cad62f5c21" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc"
I20260925 10:13:39.628882 11576 raft_consensus.cc:3037] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.628882 11716 raft_consensus.cc:3037] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.629798 11716 raft_consensus.cc:2454] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:39.629798 11576 raft_consensus.cc:2454] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:39.630018 11369 leader_election.cc:302] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.630158 11796 raft_consensus.cc:2781] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:39.630295 11796 raft_consensus.cc:686] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 LEADER]: Becoming Leader. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Running, Role: LEADER
I20260925 10:13:39.630410 11796 consensus_queue.cc:237] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } }
I20260925 10:13:39.631227 11278 catalog_manager.cc:5495] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193). New cstate: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } health_report { overall_health: HEALTHY } } }
I20260925 10:13:39.651499 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.648786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.3.152.193:37453)
0925 10:13:39.648857 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.648859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.648868 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:39.648908 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.648909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.648930 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.649001 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:39.649147 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.649741 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.650526 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.650783 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.650845 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.651081 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.651221 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.651336 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.651391 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.651425 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:13:39.653666 11716 raft_consensus.cc:1261] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.653689 11576 raft_consensus.cc:1261] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.654045 11796 consensus_queue.cc:1035] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.654204 11801 consensus_queue.cc:1035] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.656728 11806 mvcc.cc:205] Tried to move back new op lower bound from 7333196675698294784 to 7333196675606761472. Current Snapshot: MvccSnapshot[applied={T|T < 7333196675698294784}]
I20260925 10:13:39.656886 11805 mvcc.cc:205] Tried to move back new op lower bound from 7333196675698294784 to 7333196675606761472. Current Snapshot: MvccSnapshot[applied={T|T < 7333196675698294784}]
I20260925 10:13:39.659049 11278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:39.659162 11278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:39.659981 11278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Sending DeleteTablet for 3 replicas of tablet 230590e18a9440b8b52009cad62f5c21
I20260925 10:13:39.660346 11556 tablet_service.cc:1514] Processing DeleteTablet for tablet 230590e18a9440b8b52009cad62f5c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:39 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39066
I20260925 10:13:39.660349 11696 tablet_service.cc:1514] Processing DeleteTablet for tablet 230590e18a9440b8b52009cad62f5c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:39 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:58464
I20260925 10:13:39.660615 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet 230590e18a9440b8b52009cad62f5c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:39 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40664
I20260925 10:13:39.660662 11278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:39.660822 11278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:39.660856 11815 tablet_replica.cc:330] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: stopping tablet replica
I20260925 10:13:39.660948 11815 raft_consensus.cc:2229] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:39.661115 11815 raft_consensus.cc:2258] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:39.661150 11816 tablet_replica.cc:330] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: stopping tablet replica
I20260925 10:13:39.661243 11816 raft_consensus.cc:2229] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:39.661710 11815 ts_tablet_manager.cc:1905] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:39.662052 11556 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee183941ec46ec84f4d00c33147a2e (DEFAULT_TABLE table=test-table [id=184d532c1882473ea82bc8f25a18c831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.662120 11696 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee183941ec46ec84f4d00c33147a2e (DEFAULT_TABLE table=test-table [id=184d532c1882473ea82bc8f25a18c831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.662184 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee183941ec46ec84f4d00c33147a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.662238 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee183941ec46ec84f4d00c33147a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.663429 11815 ts_tablet_manager.cc:1918] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:39.663491 11815 log.cc:1196] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/230590e18a9440b8b52009cad62f5c21
I20260925 10:13:39.663637 11791 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Bootstrap starting.
I20260925 10:13:39.663776 11815 ts_tablet_manager.cc:1939] T 230590e18a9440b8b52009cad62f5c21 P 4edcef40642a487e9521625c334b8bc8: Deleting consensus metadata
I20260925 10:13:39.664041 11814 tablet_replica.cc:330] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: stopping tablet replica
I20260925 10:13:39.664083 11791 tablet_bootstrap.cc:654] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.664240 11814 raft_consensus.cc:2229] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:39.664378 11814 raft_consensus.cc:2258] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:39.664405 11792 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Bootstrap starting.
I20260925 10:13:39.664584 11816 raft_consensus.cc:2258] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:39.664783 11791 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: No bootstrap required, opened a new log
I20260925 10:13:39.664812 11792 tablet_bootstrap.cc:654] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.664829 11791 ts_tablet_manager.cc:1397] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:39.664978 11791 raft_consensus.cc:348] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.665045 11791 raft_consensus.cc:374] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.665064 11791 raft_consensus.cc:729] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.665074 11816 ts_tablet_manager.cc:1905] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:39.665094 11791 consensus_queue.cc:260] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.665174 11791 ts_tablet_manager.cc:1428] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.665238 11266 catalog_manager.cc:4841] TS 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629): tablet 230590e18a9440b8b52009cad62f5c21 (table test-table [id=07587c79d80c446785c8b588e7ded290]) successfully deleted
I20260925 10:13:39.665388 11416 tablet_service.cc:1467] Processing CreateTablet for tablet 51ee183941ec46ec84f4d00c33147a2e (DEFAULT_TABLE table=test-table [id=184d532c1882473ea82bc8f25a18c831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:39.665495 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ee183941ec46ec84f4d00c33147a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.666658 11814 ts_tablet_manager.cc:1905] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:39.666796 11790 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Bootstrap starting.
I20260925 10:13:39.667196 11790 tablet_bootstrap.cc:654] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.667907 11790 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: No bootstrap required, opened a new log
I20260925 10:13:39.667990 11790 ts_tablet_manager.cc:1397] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:39.668131 11790 raft_consensus.cc:348] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.668241 11790 raft_consensus.cc:374] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.668270 11790 raft_consensus.cc:729] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.668336 11816 ts_tablet_manager.cc:1918] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:39.668409 11816 log.cc:1196] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/230590e18a9440b8b52009cad62f5c21
I20260925 10:13:39.668326 11790 consensus_queue.cc:260] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.668501 11790 ts_tablet_manager.cc:1428] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.668685 11816 ts_tablet_manager.cc:1939] T 230590e18a9440b8b52009cad62f5c21 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting consensus metadata
I20260925 10:13:39.669068 11814 ts_tablet_manager.cc:1918] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:39.669076 11266 catalog_manager.cc:4841] TS db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453): tablet 230590e18a9440b8b52009cad62f5c21 (table test-table [id=07587c79d80c446785c8b588e7ded290]) successfully deleted
I20260925 10:13:39.669224 11814 log.cc:1196] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/230590e18a9440b8b52009cad62f5c21
I20260925 10:13:39.669484 11814 ts_tablet_manager.cc:1939] T 230590e18a9440b8b52009cad62f5c21 P 33d0c68561ed4ee39349f604a1135cbc: Deleting consensus metadata
I20260925 10:13:39.669888 11265 catalog_manager.cc:4841] TS 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905): tablet 230590e18a9440b8b52009cad62f5c21 (table test-table [id=07587c79d80c446785c8b588e7ded290]) successfully deleted
I20260925 10:13:39.670132 11792 tablet_bootstrap.cc:492] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: No bootstrap required, opened a new log
I20260925 10:13:39.670192 11792 ts_tablet_manager.cc:1397] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 10:13:39.670509 11792 raft_consensus.cc:348] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.670595 11792 raft_consensus.cc:374] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.670619 11792 raft_consensus.cc:729] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.670663 11792 consensus_queue.cc:260] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.670787 11792 ts_tablet_manager.cc:1428] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:39.792181 11798 raft_consensus.cc:482] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:39.792295 11798 raft_consensus.cc:504] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.792717 11798 leader_election.cc:288] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453), 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905)
I20260925 10:13:39.796697 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.792986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:45379 (local address 127.3.152.194:45905)
0925 10:13:39.793059 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.793061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.793071 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:39.793128 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.793129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.793142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.793184 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.793350 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.794082 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.795582 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.796246 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.796325 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.796336 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.796579 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.796623 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:13:39.796697 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.792840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:44541 (local address 127.3.152.193:37453)
0925 10:13:39.792891 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.792893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.792905 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:39.793004 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.793006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.793020 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.793083 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:39.793267 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.794021 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.795660 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.796313 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.796415 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.796424 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.796579 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.796624 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:13:39.796736 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.792788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.3.152.195:44541)
0925 10:13:39.792847 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.792860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.792911 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.793100 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.793101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.793103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.793180 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.793182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.794102 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.794103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.795474 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.795478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.796426 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.796427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.796499 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.796502 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.796564 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.796662 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":144}
I20260925 10:13:39.796746 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.792864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:45905 (local address 127.3.152.195:45379)
0925 10:13:39.792964 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.792975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.793035 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.793206 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.793207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.793208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:39.793277 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.793278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.794155 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.794155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.795474 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.795478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.796369 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.796371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.796485 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.796490 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.796564 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.796614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 10:13:39.797084 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee183941ec46ec84f4d00c33147a2e" candidate_uuid: "4edcef40642a487e9521625c334b8bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" is_pre_election: true
I20260925 10:13:39.797240 11436 raft_consensus.cc:2454] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edcef40642a487e9521625c334b8bc8 in term 0.
I20260925 10:13:39.797390 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee183941ec46ec84f4d00c33147a2e" candidate_uuid: "4edcef40642a487e9521625c334b8bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc" is_pre_election: true
I20260925 10:13:39.797483 11576 raft_consensus.cc:2454] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4edcef40642a487e9521625c334b8bc8 in term 0.
I20260925 10:13:39.797498 11650 leader_election.cc:302] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edcef40642a487e9521625c334b8bc8, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.797637 11798 raft_consensus.cc:2781] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:39.797691 11798 raft_consensus.cc:482] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:39.797722 11798 raft_consensus.cc:3037] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.798434 11798 raft_consensus.cc:504] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.798588 11798 leader_election.cc:288] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [CANDIDATE]: Term 1 election: Requested vote from peers db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453), 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905)
I20260925 10:13:39.798746 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee183941ec46ec84f4d00c33147a2e" candidate_uuid: "4edcef40642a487e9521625c334b8bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db37352c199c4cdda2bfe23e5fed4e0b"
I20260925 10:13:39.798830 11436 raft_consensus.cc:3037] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.798795 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ee183941ec46ec84f4d00c33147a2e" candidate_uuid: "4edcef40642a487e9521625c334b8bc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc"
I20260925 10:13:39.798904 11576 raft_consensus.cc:3037] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.799494 11436 raft_consensus.cc:2454] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edcef40642a487e9521625c334b8bc8 in term 1.
I20260925 10:13:39.799633 11576 raft_consensus.cc:2454] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4edcef40642a487e9521625c334b8bc8 in term 1.
I20260925 10:13:39.799660 11650 leader_election.cc:302] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4edcef40642a487e9521625c334b8bc8, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.799757 11798 raft_consensus.cc:2781] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:39.799880 11798 raft_consensus.cc:686] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Running, Role: LEADER
I20260925 10:13:39.799929 11798 consensus_queue.cc:237] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.800691 11279 catalog_manager.cc:5495] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4edcef40642a487e9521625c334b8bc8 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "4edcef40642a487e9521625c334b8bc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } health_report { overall_health: HEALTHY } } }
Sep 25 10:13:39 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331218, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.254@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:39.840694 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.832954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.3.152.254:41947)
0925 10:13:39.833047 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.833050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.833190 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:13:39.833290 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.833291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.833303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.833374 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:39.833910 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.834492 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.835285 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.835488 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.835578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.837911 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:39.837913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:39.837913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:39.837922 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:39.839239 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.839718 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.839719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.839720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.839771 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:39.840002 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:39.840003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:39.840004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:39.840139 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:39.840218 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.840442 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.840554 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:13:39.840761 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.832858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41947 (local address 127.0.0.1:45432)
0925 10:13:39.833078 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.833096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.833215 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.833396 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.833397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.833756 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:39.833807 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.833809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.834584 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.834585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.835192 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.835194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.835579 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.835580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.835658 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.836224 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:39.836229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:39.837812 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:39.839345 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.839354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.839357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.839579 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.839842 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:39.839843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:39.839843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:39.839909 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:39.840243 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:39.840251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:39.840316 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.840416 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.840621 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":86,"thread_start_us":120,"threads_started":1}
I20260925 10:13:39.842890 11279 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:45432:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 10:13:39.843087 11279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:39.844741 11696 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb33b6443d24084beec4205f6e81ad6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0fa8bdae533416a90e73e25cb984c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:13:39.844741 11556 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb33b6443d24084beec4205f6e81ad6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0fa8bdae533416a90e73e25cb984c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:13:39.844743 11416 tablet_service.cc:1467] Processing CreateTablet for tablet 4fb33b6443d24084beec4205f6e81ad6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0fa8bdae533416a90e73e25cb984c57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 10:13:39.844913 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb33b6443d24084beec4205f6e81ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.844913 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb33b6443d24084beec4205f6e81ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.844933 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fb33b6443d24084beec4205f6e81ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:39.846148 11790 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: Bootstrap starting.
I20260925 10:13:39.846427 11791 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: Bootstrap starting.
I20260925 10:13:39.846427 11792 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: Bootstrap starting.
I20260925 10:13:39.846514 11790 tablet_bootstrap.cc:654] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.846844 11791 tablet_bootstrap.cc:654] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.846844 11792 tablet_bootstrap.cc:654] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:39.847543 11791 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: No bootstrap required, opened a new log
I20260925 10:13:39.847667 11791 ts_tablet_manager.cc:1397] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:39.847723 11790 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: No bootstrap required, opened a new log
I20260925 10:13:39.847779 11790 ts_tablet_manager.cc:1397] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:39.847839 11791 raft_consensus.cc:348] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.847898 11790 raft_consensus.cc:348] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.847946 11791 raft_consensus.cc:374] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.847992 11790 raft_consensus.cc:374] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.848003 11791 raft_consensus.cc:729] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.848029 11790 raft_consensus.cc:729] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.848083 11790 consensus_queue.cc:260] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.848053 11791 consensus_queue.cc:260] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.848192 11790 ts_tablet_manager.cc:1428] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:39.848205 11791 ts_tablet_manager.cc:1428] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:39.848235 11801 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.848340 11801 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.848440 11792 tablet_bootstrap.cc:492] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: No bootstrap required, opened a new log
I20260925 10:13:39.848489 11792 ts_tablet_manager.cc:1397] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:13:39.848469 11798 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.848541 11798 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.848624 11792 raft_consensus.cc:348] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.848702 11792 raft_consensus.cc:374] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:39.848726 11792 raft_consensus.cc:729] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Initialized, Role: FOLLOWER
I20260925 10:13:39.848768 11792 consensus_queue.cc:260] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.848868 11792 ts_tablet_manager.cc:1428] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:39.848929 11797 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.849059 11797 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.948772 11801 raft_consensus.cc:482] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:39.948871 11801 raft_consensus.cc:504] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.949043 11801 leader_election.cc:288] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:39.949268 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb33b6443d24084beec4205f6e81ad6" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8" is_pre_election: true
I20260925 10:13:39.949255 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb33b6443d24084beec4205f6e81ad6" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc" is_pre_election: true
I20260925 10:13:39.949354 11716 raft_consensus.cc:2454] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:39.949376 11576 raft_consensus.cc:2454] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:39.949537 11369 leader_election.cc:302] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.949661 11801 raft_consensus.cc:2781] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:39.949762 11801 raft_consensus.cc:482] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:39.949795 11801 raft_consensus.cc:3037] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.950562 11801 raft_consensus.cc:504] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.950758 11801 leader_election.cc:288] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Requested vote from peers 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:39.950960 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb33b6443d24084beec4205f6e81ad6" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc"
I20260925 10:13:39.950953 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fb33b6443d24084beec4205f6e81ad6" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8"
I20260925 10:13:39.951038 11716 raft_consensus.cc:3037] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.951038 11576 raft_consensus.cc:3037] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:39.951817 11716 raft_consensus.cc:2454] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:39.951817 11576 raft_consensus.cc:2454] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:39.952045 11369 leader_election.cc:302] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:39.952198 11801 raft_consensus.cc:2781] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:39.952262 11801 raft_consensus.cc:686] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 LEADER]: Becoming Leader. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Running, Role: LEADER
I20260925 10:13:39.952324 11801 consensus_queue.cc:237] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:39.952644 11809 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: TxnStatusTablet state changed. Reason: New leader db37352c199c4cdda2bfe23e5fed4e0b. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.952737 11809 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:13:39.952950 11833 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 10:13:39.953078 11833 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 10:13:39.953034 11278 catalog_manager.cc:5495] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193). New cstate: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260925 10:13:39.967406 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.964751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.3.152.193:37453)
0925 10:13:39.964803 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.964806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.964883 (+ 77us) server_negotiation.cc:407] Connection header received
0925 10:13:39.964972 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.964973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.964981 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.965027 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:39.965162 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.965638 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.966762 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.966943 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.966992 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.967179 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.967206 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.967244 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.967327 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.967345 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:13:39.967463 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.964748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.0.0.1:40684)
0925 10:13:39.964820 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.964833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.964896 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.965044 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.965045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.965046 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:13:39.965104 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.965107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.965708 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.965709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.966647 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.966650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.967011 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.967012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.967075 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.967097 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.967261 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.967269 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.967323 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.967377 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 10:13:39.968457 11716 raft_consensus.cc:1261] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.968472 11576 raft_consensus.cc:1261] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.968711 11801 consensus_queue.cc:1035] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.968842 11809 consensus_queue.cc:1035] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.969451 11797 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: TxnStatusTablet state changed. Reason: New leader db37352c199c4cdda2bfe23e5fed4e0b. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.969463 11798 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: TxnStatusTablet state changed. Reason: New leader db37352c199c4cdda2bfe23e5fed4e0b. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.969548 11797 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.969554 11798 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.970207 11809 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.970297 11809 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:13:39.970376 11804 mvcc.cc:205] Tried to move back new op lower bound from 7333196676989173760 to 7333196676925116416. Current Snapshot: MvccSnapshot[applied={T|T < 7333196676989173760}]
I20260925 10:13:39.970988 11798 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.971082 11798 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 4edcef40642a487e9521625c334b8bc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.971195 11801 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.971274 11801 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P db37352c199c4cdda2bfe23e5fed4e0b: This TxnStatusTablet replica's current role is: LEADER
I20260925 10:13:39.972045 11797 tablet_replica.cc:439] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } } }
I20260925 10:13:39.972157 11797 tablet_replica.cc:442] T 4fb33b6443d24084beec4205f6e81ad6 P 33d0c68561ed4ee39349f604a1135cbc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 10:13:39.974936 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.972431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.3.152.195:46629)
0925 10:13:39.972534 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.972536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.972545 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:39.972581 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.972582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.972603 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.972671 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:39.972801 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.973278 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.974078 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.974291 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.974366 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.974541 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.974667 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.974755 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.974810 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.974830 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:13:39.982713 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.977752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:54391 (local address 127.3.152.193:37453)
0925 10:13:39.977866 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.977868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.980201 (+ 2333us) server_negotiation.cc:407] Connection header received
0925 10:13:39.980270 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.980271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.980285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.980353 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:39.980497 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.981087 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.981999 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.982198 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.982269 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.982449 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.982488 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.982546 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.982609 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.982630 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 10:13:39.982923 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.977626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.3.152.195:54391)
0925 10:13:39.980106 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.980127 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.980216 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.980356 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.980357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.980358 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:13:39.980432 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.980433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.981205 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.981206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.981884 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.981887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.982280 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.982282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.982350 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.982363 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.982549 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.982552 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.982604 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.982645 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2423,"thread_start_us":71,"threads_started":1}
I20260925 10:13:39.985204 11576 raft_consensus.cc:1261] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Refusing update from remote peer 4edcef40642a487e9521625c334b8bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.985450 11436 raft_consensus.cc:1261] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Refusing update from remote peer 4edcef40642a487e9521625c334b8bc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:39.985613 11821 consensus_queue.cc:1035] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.985728 11798 consensus_queue.cc:1035] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:39.995360 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.992322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46629 (local address 127.3.152.193:47179)
0925 10:13:39.992471 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:39.992484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:39.992549 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:39.992676 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:39.992677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:39.992678 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:13:39.992731 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.992733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.993447 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.993449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.994387 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:39.994390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.994931 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.994932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:39.994995 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.995015 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.995161 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.995163 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:13:39.995215 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:13:39.995264 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 10:13:39.995419 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:39.992376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:47179 (local address 127.3.152.195:46629)
0925 10:13:39.992421 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:39.992423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:39.992534 (+ 111us) server_negotiation.cc:407] Connection header received
0925 10:13:39.992602 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:39.992603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:39.992617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:39.992673 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:39.992888 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.993370 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.994527 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:39.994860 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:39.994922 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:39.995095 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:39.995122 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:39.995160 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:39.995292 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 10:13:39.995315 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:13:40.008613 11278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.008740 11278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.009413 11278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Sending DeleteTablet for 3 replicas of tablet 51ee183941ec46ec84f4d00c33147a2e
I20260925 10:13:40.009742 11556 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee183941ec46ec84f4d00c33147a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39066
I20260925 10:13:40.009779 11696 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee183941ec46ec84f4d00c33147a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:58464
I20260925 10:13:40.009868 11815 tablet_replica.cc:330] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: stopping tablet replica
I20260925 10:13:40.009867 11814 tablet_replica.cc:330] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: stopping tablet replica
I20260925 10:13:40.009922 11815 raft_consensus.cc:2229] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:40.009922 11814 raft_consensus.cc:2229] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.009967 11814 raft_consensus.cc:2258] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.010013 11815 raft_consensus.cc:2258] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.010334 11814 ts_tablet_manager.cc:1905] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.012127 11814 ts_tablet_manager.cc:1918] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:13:40.012182 11814 log.cc:1196] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/51ee183941ec46ec84f4d00c33147a2e
I20260925 10:13:40.012404 11814 ts_tablet_manager.cc:1939] T 51ee183941ec46ec84f4d00c33147a2e P 33d0c68561ed4ee39349f604a1135cbc: Deleting consensus metadata
I20260925 10:13:40.012869 11815 ts_tablet_manager.cc:1905] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.013743 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet 51ee183941ec46ec84f4d00c33147a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40664
I20260925 10:13:40.013856 11816 tablet_replica.cc:330] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: stopping tablet replica
I20260925 10:13:40.013916 11816 raft_consensus.cc:2229] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.013972 11816 raft_consensus.cc:2258] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.014385 11816 ts_tablet_manager.cc:1905] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.014652 11815 ts_tablet_manager.cc:1918] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:13:40.014725 11815 log.cc:1196] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/51ee183941ec46ec84f4d00c33147a2e
I20260925 10:13:40.014943 11815 ts_tablet_manager.cc:1939] T 51ee183941ec46ec84f4d00c33147a2e P 4edcef40642a487e9521625c334b8bc8: Deleting consensus metadata
I20260925 10:13:40.015399 11265 catalog_manager.cc:4841] TS 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905): tablet 51ee183941ec46ec84f4d00c33147a2e (table test-table [id=184d532c1882473ea82bc8f25a18c831]) successfully deleted
I20260925 10:13:40.015735 11266 catalog_manager.cc:4841] TS 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629): tablet 51ee183941ec46ec84f4d00c33147a2e (table test-table [id=184d532c1882473ea82bc8f25a18c831]) successfully deleted
I20260925 10:13:40.016146 11816 ts_tablet_manager.cc:1918] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:13:40.016206 11816 log.cc:1196] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/51ee183941ec46ec84f4d00c33147a2e
I20260925 10:13:40.016413 11816 ts_tablet_manager.cc:1939] T 51ee183941ec46ec84f4d00c33147a2e P db37352c199c4cdda2bfe23e5fed4e0b: Deleting consensus metadata
I20260925 10:13:40.016839 11266 catalog_manager.cc:4841] TS db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453): tablet 51ee183941ec46ec84f4d00c33147a2e (table test-table [id=184d532c1882473ea82bc8f25a18c831]) successfully deleted
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:13:40.049772 11329 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:40.042454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.3.152.254:41947)
0925 10:13:40.042515 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.042517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.042697 (+ 180us) server_negotiation.cc:407] Connection header received
0925 10:13:40.042832 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.042833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.042849 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.042885 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:40.043722 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.044544 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.045657 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.045983 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.046059 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.049699 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45446: BlockingRecv error: recv got EOF from 127.0.0.1:45446 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM
I20260925 10:13:40.060103 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.050666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.3.152.254:41947)
0925 10:13:40.050714 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.050715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.050853 (+ 138us) server_negotiation.cc:407] Connection header received
0925 10:13:40.050924 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.050925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.050936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.050999 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:40.051597 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.052462 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.053465 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.053796 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.053869 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.056778 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:40.056780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:40.056781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:40.056795 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:40.058658 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:40.059120 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:40.059121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:40.059122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:40.059195 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:40.059439 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:40.059440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:40.059441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:40.059597 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:40.059717 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.059851 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.059942 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:13:40.062225 11278 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:13:40.062419 11278 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:40.063910 11556 tablet_service.cc:1467] Processing CreateTablet for tablet a386e351b428419b9e9fc64295dcf546 (DEFAULT_TABLE table=test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.064083 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a386e351b428419b9e9fc64295dcf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.064224 11696 tablet_service.cc:1467] Processing CreateTablet for tablet a386e351b428419b9e9fc64295dcf546 (DEFAULT_TABLE table=test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.064357 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a386e351b428419b9e9fc64295dcf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.065488 11792 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Bootstrap starting.
I20260925 10:13:40.065748 11791 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Bootstrap starting.
I20260925 10:13:40.065936 11792 tablet_bootstrap.cc:654] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.066181 11791 tablet_bootstrap.cc:654] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.066303 11416 tablet_service.cc:1467] Processing CreateTablet for tablet a386e351b428419b9e9fc64295dcf546 (DEFAULT_TABLE table=test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.066421 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a386e351b428419b9e9fc64295dcf546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.067843 11790 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Bootstrap starting.
I20260925 10:13:40.068225 11790 tablet_bootstrap.cc:654] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.068442 11792 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: No bootstrap required, opened a new log
I20260925 10:13:40.068488 11792 ts_tablet_manager.cc:1397] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 10:13:40.068652 11792 raft_consensus.cc:348] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.068745 11792 raft_consensus.cc:374] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.068766 11792 raft_consensus.cc:729] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.068811 11792 consensus_queue.cc:260] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.068897 11792 ts_tablet_manager.cc:1428] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:40.069574 11790 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: No bootstrap required, opened a new log
I20260925 10:13:40.069628 11790 ts_tablet_manager.cc:1397] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:13:40.069764 11790 raft_consensus.cc:348] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.069854 11790 raft_consensus.cc:374] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.069876 11790 raft_consensus.cc:729] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.069928 11790 consensus_queue.cc:260] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.070025 11790 ts_tablet_manager.cc:1428] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:40.072439 11791 tablet_bootstrap.cc:492] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: No bootstrap required, opened a new log
I20260925 10:13:40.072503 11791 ts_tablet_manager.cc:1397] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260925 10:13:40.072664 11791 raft_consensus.cc:348] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.072746 11791 raft_consensus.cc:374] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.072767 11791 raft_consensus.cc:729] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.072817 11791 consensus_queue.cc:260] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.072899 11791 ts_tablet_manager.cc:1428] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:40.236685 11809 raft_consensus.cc:482] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:40.236814 11809 raft_consensus.cc:504] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.237020 11809 leader_election.cc:288] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:40.237313 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a386e351b428419b9e9fc64295dcf546" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc" is_pre_election: true
I20260925 10:13:40.237316 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a386e351b428419b9e9fc64295dcf546" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8" is_pre_election: true
I20260925 10:13:40.237407 11576 raft_consensus.cc:2454] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:40.237435 11716 raft_consensus.cc:2454] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 0.
I20260925 10:13:40.237584 11369 leader_election.cc:302] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:40.237684 11809 raft_consensus.cc:2781] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:40.237749 11809 raft_consensus.cc:482] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:40.237783 11809 raft_consensus.cc:3037] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.238555 11809 raft_consensus.cc:504] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.238736 11809 leader_election.cc:288] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Requested vote from peers 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:40.238966 11576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a386e351b428419b9e9fc64295dcf546" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33d0c68561ed4ee39349f604a1135cbc"
I20260925 10:13:40.239001 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a386e351b428419b9e9fc64295dcf546" candidate_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8"
I20260925 10:13:40.239066 11576 raft_consensus.cc:3037] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.239081 11716 raft_consensus.cc:3037] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.239727 11576 raft_consensus.cc:2454] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:40.239840 11716 raft_consensus.cc:2454] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db37352c199c4cdda2bfe23e5fed4e0b in term 1.
I20260925 10:13:40.239951 11369 leader_election.cc:302] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:40.240083 11809 raft_consensus.cc:2781] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:40.240170 11809 raft_consensus.cc:686] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 LEADER]: Becoming Leader. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Running, Role: LEADER
I20260925 10:13:40.240236 11809 consensus_queue.cc:237] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.240942 11278 catalog_manager.cc:5495] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193). New cstate: current_term: 1 leader_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260925 10:13:40.253152 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.250451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.3.152.193:37453)
0925 10:13:40.250545 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.250548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.250558 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:40.250598 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.250599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.250612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.250692 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:40.250873 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.251399 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.252362 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.252580 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.252669 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.252838 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.252888 (+ 50us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.252963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.253031 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.253054 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 10:13:40.254150 11716 raft_consensus.cc:1261] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:40.254276 11576 raft_consensus.cc:1261] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Refusing update from remote peer db37352c199c4cdda2bfe23e5fed4e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:40.254418 11796 consensus_queue.cc:1035] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:40.254617 11809 consensus_queue.cc:1035] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:40.258023 11279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.258137 11279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.258801 11279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Sending DeleteTablet for 3 replicas of tablet a386e351b428419b9e9fc64295dcf546
I20260925 10:13:40.259088 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet a386e351b428419b9e9fc64295dcf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40664
I20260925 10:13:40.259110 11696 tablet_service.cc:1514] Processing DeleteTablet for tablet a386e351b428419b9e9fc64295dcf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:58464
I20260925 10:13:40.259204 11816 tablet_replica.cc:330] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: stopping tablet replica
I20260925 10:13:40.259258 11816 raft_consensus.cc:2229] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:40.259231 11556 tablet_service.cc:1514] Processing DeleteTablet for tablet a386e351b428419b9e9fc64295dcf546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39066
I20260925 10:13:40.259318 11816 raft_consensus.cc:2258] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.259342 11815 tablet_replica.cc:330] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: stopping tablet replica
I20260925 10:13:40.259419 11815 raft_consensus.cc:2229] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.259358 11279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 10:13:40.259486 11815 raft_consensus.cc:2258] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 10:13:40.259546 11279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:13:40.259574 11814 tablet_replica.cc:330] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: stopping tablet replica
I20260925 10:13:40.259635 11814 raft_consensus.cc:2229] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.259691 11814 raft_consensus.cc:2258] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.260082 11814 ts_tablet_manager.cc:1905] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.260252 11815 ts_tablet_manager.cc:1905] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.260257 11816 ts_tablet_manager.cc:1905] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.260869 11556 tablet_service.cc:1467] Processing CreateTablet for tablet 94953a35a075412491ef0d10a13f5931 (DEFAULT_TABLE table=test-table [id=954d626c9c384324aebf80c51322f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.260987 11556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94953a35a075412491ef0d10a13f5931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.262058 11814 ts_tablet_manager.cc:1918] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:40.262060 11815 ts_tablet_manager.cc:1918] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:40.262077 11816 ts_tablet_manager.cc:1918] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:13:40.262138 11816 log.cc:1196] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/a386e351b428419b9e9fc64295dcf546
I20260925 10:13:40.262132 11815 log.cc:1196] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/a386e351b428419b9e9fc64295dcf546
I20260925 10:13:40.262132 11814 log.cc:1196] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/a386e351b428419b9e9fc64295dcf546
I20260925 10:13:40.262372 11816 ts_tablet_manager.cc:1939] T a386e351b428419b9e9fc64295dcf546 P db37352c199c4cdda2bfe23e5fed4e0b: Deleting consensus metadata
I20260925 10:13:40.262420 11814 ts_tablet_manager.cc:1939] T a386e351b428419b9e9fc64295dcf546 P 33d0c68561ed4ee39349f604a1135cbc: Deleting consensus metadata
I20260925 10:13:40.262430 11815 ts_tablet_manager.cc:1939] T a386e351b428419b9e9fc64295dcf546 P 4edcef40642a487e9521625c334b8bc8: Deleting consensus metadata
I20260925 10:13:40.262501 11792 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Bootstrap starting.
I20260925 10:13:40.262890 11792 tablet_bootstrap.cc:654] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.263301 11265 catalog_manager.cc:4841] TS 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905): tablet a386e351b428419b9e9fc64295dcf546 (table test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]) successfully deleted
I20260925 10:13:40.263552 11266 catalog_manager.cc:4841] TS 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629): tablet a386e351b428419b9e9fc64295dcf546 (table test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]) successfully deleted
I20260925 10:13:40.263626 11266 catalog_manager.cc:4841] TS db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453): tablet a386e351b428419b9e9fc64295dcf546 (table test-table [id=8f7fbe1b6faf4aa2b7157b8d49da266b]) successfully deleted
I20260925 10:13:40.263715 11696 tablet_service.cc:1467] Processing CreateTablet for tablet 94953a35a075412491ef0d10a13f5931 (DEFAULT_TABLE table=test-table [id=954d626c9c384324aebf80c51322f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.263715 11416 tablet_service.cc:1467] Processing CreateTablet for tablet 94953a35a075412491ef0d10a13f5931 (DEFAULT_TABLE table=test-table [id=954d626c9c384324aebf80c51322f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:13:40.263808 11416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94953a35a075412491ef0d10a13f5931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.263809 11696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94953a35a075412491ef0d10a13f5931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:40.264986 11790 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: Bootstrap starting.
I20260925 10:13:40.265017 11791 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Bootstrap starting.
I20260925 10:13:40.265290 11790 tablet_bootstrap.cc:654] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.265332 11791 tablet_bootstrap.cc:654] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:40.265810 11790 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: No bootstrap required, opened a new log
I20260925 10:13:40.265861 11790 ts_tablet_manager.cc:1397] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 10:13:40.265956 11791 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: No bootstrap required, opened a new log
I20260925 10:13:40.265990 11791 ts_tablet_manager.cc:1397] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:13:40.266001 11790 raft_consensus.cc:348] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.266068 11790 raft_consensus.cc:374] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.266088 11790 raft_consensus.cc:729] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db37352c199c4cdda2bfe23e5fed4e0b, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.266088 11791 raft_consensus.cc:348] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.266161 11791 raft_consensus.cc:374] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.266137 11790 consensus_queue.cc:260] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.266183 11791 raft_consensus.cc:729] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edcef40642a487e9521625c334b8bc8, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.266239 11790 ts_tablet_manager.cc:1428] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:40.266227 11791 consensus_queue.cc:260] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.266316 11791 ts_tablet_manager.cc:1428] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:40.266570 11792 tablet_bootstrap.cc:492] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: No bootstrap required, opened a new log
I20260925 10:13:40.266646 11792 ts_tablet_manager.cc:1397] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 10:13:40.266809 11792 raft_consensus.cc:348] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.266885 11792 raft_consensus.cc:374] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:40.266911 11792 raft_consensus.cc:729] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Initialized, Role: FOLLOWER
I20260925 10:13:40.266954 11792 consensus_queue.cc:260] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.267043 11792 ts_tablet_manager.cc:1428] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:40.419536 11797 raft_consensus.cc:482] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:13:40.419715 11797 raft_consensus.cc:504] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.420099 11797 leader_election.cc:288] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:40.423347 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.420325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.3.152.194:41405)
0925 10:13:40.420495 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:40.420510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:40.420701 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:40.420884 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:40.420885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:40.420887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:40.420963 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.420966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.421693 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.421694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.422469 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.422472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.423041 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.423042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.423123 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.423127 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:40.423199 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 10:13:40.423246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260925 10:13:40.423337 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.420429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:41405 (local address 127.3.152.193:37453)
0925 10:13:40.420501 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.420504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.420604 (+ 100us) server_negotiation.cc:407] Connection header received
0925 10:13:40.420790 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.420791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.420803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.420867 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:40.421035 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.421509 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.422564 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.422943 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.423028 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.423042 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.423207 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.423251 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:13:40.423337 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.420624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42765 (local address 127.3.152.195:46629)
0925 10:13:40.420685 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.420687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.420698 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:40.420786 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.420787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.420796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.420857 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:13:40.421011 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.421501 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.422518 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.422890 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.422972 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.422981 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.423207 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.423260 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:13:40.423372 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.420542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46629 (local address 127.3.152.194:42765)
0925 10:13:40.420619 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:40.420629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:40.420690 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:40.420878 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:40.420879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:40.420882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:13:40.420940 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.420942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.421619 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.421620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.422414 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.422416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.422982 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.422983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.423049 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.423058 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 10:13:40.423192 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:13:40.423261 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":38,"threads_started":1}
I20260925 10:13:40.423854 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94953a35a075412491ef0d10a13f5931" candidate_uuid: "33d0c68561ed4ee39349f604a1135cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" is_pre_election: true
I20260925 10:13:40.423877 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94953a35a075412491ef0d10a13f5931" candidate_uuid: "33d0c68561ed4ee39349f604a1135cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8" is_pre_election: true
I20260925 10:13:40.423981 11436 raft_consensus.cc:2454] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d0c68561ed4ee39349f604a1135cbc in term 0.
I20260925 10:13:40.424014 11716 raft_consensus.cc:2454] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d0c68561ed4ee39349f604a1135cbc in term 0.
I20260925 10:13:40.424201 11510 leader_election.cc:302] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:40.424350 11797 raft_consensus.cc:2781] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:13:40.424410 11797 raft_consensus.cc:482] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:13:40.424440 11797 raft_consensus.cc:3037] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.425130 11797 raft_consensus.cc:504] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.425272 11797 leader_election.cc:288] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [CANDIDATE]: Term 1 election: Requested vote from peers db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453), 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629)
I20260925 10:13:40.425547 11716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94953a35a075412491ef0d10a13f5931" candidate_uuid: "33d0c68561ed4ee39349f604a1135cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4edcef40642a487e9521625c334b8bc8"
I20260925 10:13:40.425547 11436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94953a35a075412491ef0d10a13f5931" candidate_uuid: "33d0c68561ed4ee39349f604a1135cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db37352c199c4cdda2bfe23e5fed4e0b"
I20260925 10:13:40.425639 11436 raft_consensus.cc:3037] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.425639 11716 raft_consensus.cc:3037] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:40.426425 11716 raft_consensus.cc:2454] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d0c68561ed4ee39349f604a1135cbc in term 1.
I20260925 10:13:40.426425 11436 raft_consensus.cc:2454] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d0c68561ed4ee39349f604a1135cbc in term 1.
I20260925 10:13:40.426630 11510 leader_election.cc:302] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d0c68561ed4ee39349f604a1135cbc, db37352c199c4cdda2bfe23e5fed4e0b; no voters:
I20260925 10:13:40.426726 11797 raft_consensus.cc:2781] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:40.426867 11797 raft_consensus.cc:686] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 1 LEADER]: Becoming Leader. State: Replica: 33d0c68561ed4ee39349f604a1135cbc, State: Running, Role: LEADER
I20260925 10:13:40.426993 11797 consensus_queue.cc:237] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } }
I20260925 10:13:40.427712 11278 catalog_manager.cc:5495] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194). New cstate: current_term: 1 leader_uuid: "33d0c68561ed4ee39349f604a1135cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d0c68561ed4ee39349f604a1135cbc" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260925 10:13:40.454826 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.452348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.3.152.194:45905)
0925 10:13:40.452556 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.452558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.452566 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:13:40.452636 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.452637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.452658 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.452739 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:40.452848 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.453407 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.454100 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.454310 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.454372 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.454487 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.454597 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.454655 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.454675 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.454693 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":128,"threads_started":1}
I20260925 10:13:40.459656 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.457329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:43537 (local address 127.3.152.193:37453)
0925 10:13:40.457375 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.457377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.457396 (+ 19us) server_negotiation.cc:407] Connection header received
0925 10:13:40.457448 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.457449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.457472 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.457538 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:40.457687 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.458159 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.458935 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.459141 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.459220 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.459398 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.459436 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.459492 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.459551 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.459572 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:13:40.459695 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.457162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.3.152.194:43537)
0925 10:13:40.457336 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:40.457349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:40.457410 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:40.457545 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:40.457546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:40.457548 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:13:40.457614 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.457615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.458244 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.458245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.458841 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.458843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.459226 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.459227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.459296 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.459309 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.459495 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.459497 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:13:40.459538 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 10:13:40.459581 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260925 10:13:40.461790 11716 raft_consensus.cc:1261] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Refusing update from remote peer 33d0c68561ed4ee39349f604a1135cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:40.461833 11436 raft_consensus.cc:1261] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Refusing update from remote peer 33d0c68561ed4ee39349f604a1135cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:13:40.462078 11797 consensus_queue.cc:1035] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db37352c199c4cdda2bfe23e5fed4e0b" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 37453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:40.462226 11871 consensus_queue.cc:1035] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4edcef40642a487e9521625c334b8bc8" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:13:40.469122 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.466828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:52673 (local address 127.3.152.194:45905)
0925 10:13:40.466864 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:40.466866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:40.466878 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:40.466949 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:40.466950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:40.466961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:40.467037 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:13:40.467168 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.467637 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.468411 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.468660 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.468730 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.468899 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.468931 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.468977 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:40.469034 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 10:13:40.469053 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:13:40.469148 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.466790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:45905 (local address 127.3.152.193:52673)
0925 10:13:40.466841 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:40.466853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:40.466894 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:40.467046 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:40.467047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:40.467049 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 10:13:40.467105 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.467106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.467727 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.467728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.468335 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:40.468337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:40.468750 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:40.468752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:40.468813 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:40.468819 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 10:13:40.468976 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 10:13:40.468978 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:13:40.469029 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:13:40.469077 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 10:13:40.475562 11278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.475692 11278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:13:40.476387 11278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d641a34d07214e839df9740913b44bcd: Sending DeleteTablet for 3 replicas of tablet 94953a35a075412491ef0d10a13f5931
I20260925 10:13:40.476650 11556 tablet_service.cc:1514] Processing DeleteTablet for tablet 94953a35a075412491ef0d10a13f5931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39066
I20260925 10:13:40.476756 11814 tablet_replica.cc:330] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: stopping tablet replica
I20260925 10:13:40.476747 11416 tablet_service.cc:1514] Processing DeleteTablet for tablet 94953a35a075412491ef0d10a13f5931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40664
I20260925 10:13:40.476804 11814 raft_consensus.cc:2229] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:13:40.476864 11696 tablet_service.cc:1514] Processing DeleteTablet for tablet 94953a35a075412491ef0d10a13f5931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:13:40 UTC) from {username='oryx', principal='oryx/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:58464
I20260925 10:13:40.476888 11814 raft_consensus.cc:2258] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.476948 11816 tablet_replica.cc:330] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b: stopping tablet replica
I20260925 10:13:40.477003 11816 raft_consensus.cc:2229] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.477038 11816 raft_consensus.cc:2258] T 94953a35a075412491ef0d10a13f5931 P db37352c199c4cdda2bfe23e5fed4e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.477032 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 11351
I20260925 10:13:40.477102 11815 tablet_replica.cc:330] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: stopping tablet replica
I20260925 10:13:40.477155 11815 raft_consensus.cc:2229] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:13:40.477191 11815 raft_consensus.cc:2258] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:13:40.477231 11814 ts_tablet_manager.cc:1905] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.478909 11814 ts_tablet_manager.cc:1918] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:13:40.478965 11814 log.cc:1196] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/94953a35a075412491ef0d10a13f5931
I20260925 10:13:40.479198 11814 ts_tablet_manager.cc:1939] T 94953a35a075412491ef0d10a13f5931 P 33d0c68561ed4ee39349f604a1135cbc: Deleting consensus metadata
I20260925 10:13:40.479242 11815 ts_tablet_manager.cc:1905] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:13:40.479692 11265 catalog_manager.cc:4841] TS 33d0c68561ed4ee39349f604a1135cbc (127.3.152.194:45905): tablet 94953a35a075412491ef0d10a13f5931 (table test-table [id=954d626c9c384324aebf80c51322f19f]) successfully deleted
I20260925 10:13:40.481186 11815 ts_tablet_manager.cc:1918] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 10:13:40.481258 11815 log.cc:1196] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/94953a35a075412491ef0d10a13f5931
I20260925 10:13:40.481467 11815 ts_tablet_manager.cc:1939] T 94953a35a075412491ef0d10a13f5931 P 4edcef40642a487e9521625c334b8bc8: Deleting consensus metadata
I20260925 10:13:40.481899 11266 catalog_manager.cc:4841] TS 4edcef40642a487e9521625c334b8bc8 (127.3.152.195:46629): tablet 94953a35a075412491ef0d10a13f5931 (table test-table [id=954d626c9c384324aebf80c51322f19f]) successfully deleted
W20260925 10:13:40.485347 11266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:37453 (error 108)
I20260925 10:13:40.485668 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 11491
I20260925 10:13:40.486413 11882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:40.485799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:37453 (local address 127.0.0.1:40702)
0925 10:13:40.486320 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:40.486367 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:37453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":41,"threads_started":1}
W20260925 10:13:40.486521 11266 catalog_manager.cc:4584] TS db37352c199c4cdda2bfe23e5fed4e0b (127.3.152.193:37453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94953a35a075412491ef0d10a13f5931: Network error: Client connection negotiation failed: client connection to 127.3.152.193:37453: connect: Connection refused (error 111)
I20260925 10:13:40.492137 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 11631
I20260925 10:13:40.498556 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 11247
2026-09-25T10:13:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3026 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11885](info): set up 2 sockets
Sep 25 10:13:40 dist-test-slave-dzwq krb5kdc[11885](info): commencing operation
krb5kdc: starting...
W20260925 10:13:41.581307 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:13:41 dist-test-slave-dzwq krb5kdc[11885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:13:41.707314 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38837
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:41.791046 11898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:41.791316 11898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:41.791356 11898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:41.792629 11898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:41.792701 11898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:41.792726 11898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:41.792748 11898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:41.792776 11898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:41.794348 11898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:41.794646 11898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:41.794883 11898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:13:41.800454 11898 server_base.cc:1034] running on GCE node
W20260925 10:13:41.800511 11904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:41.800434 11906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:41.800612 11903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:41.800911 11898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:41.801221 11898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:41.802399 11898 hybrid_clock.cc:648] HybridClock initialized: now 1790331221802395 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T10:13:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1304 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:13:41 dist-test-slave-dzwq krb5kdc[11911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:13:41 dist-test-slave-dzwq krb5kdc[11911](info): set up 2 sockets
Sep 25 10:13:41 dist-test-slave-dzwq krb5kdc[11911](info): commencing operation
krb5kdc: starting...
W20260925 10:13:42.849212 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:13:42 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:13:42.975761 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:32851
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:43.057492 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.057770 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.057812 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.059046 11924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:43.059129 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.059152 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.059173 11924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:43.059204 11924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:43.060771 11924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.061071 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.061309 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.066373 11929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.066380 11930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.066383 11932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.066617 11924 server_base.cc:1034] running on GCE node
I20260925 10:13:43.066892 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.067219 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.068384 11924 hybrid_clock.cc:648] HybridClock initialized: now 1790331223068369 us; error 43 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.074630 11924 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:13:43.075132 11924 webserver.cc:460] Webserver started at http://127.3.152.254:46427/ using document root <none> and password file <none>
I20260925 10:13:43.075379 11924 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.075448 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.075598 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:43.076818 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0bc48e05b61a4f00954f1103e8a8aa58"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "8eddb36d337115cb682e9980bd8829d6"
server_key_iv: "85e6cf2e970a6acc1298a9b8ffe9f7b8"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.077198 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0bc48e05b61a4f00954f1103e8a8aa58"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "8eddb36d337115cb682e9980bd8829d6"
server_key_iv: "85e6cf2e970a6acc1298a9b8ffe9f7b8"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.078608 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:13:43.079416 11939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.079638 11924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:43.079746 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0bc48e05b61a4f00954f1103e8a8aa58"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "8eddb36d337115cb682e9980bd8829d6"
server_key_iv: "85e6cf2e970a6acc1298a9b8ffe9f7b8"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.079820 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:43.092536 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:43.094795 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:43.094990 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:43.099141 11924 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:32851
I20260925 10:13:43.099170 11991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:32851 every 8 connection(s)
I20260925 10:13:43.099534 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:43.100464 11992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:43.101138 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 11924
I20260925 10:13:43.101248 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:43.101451 3683 external_mini_cluster.cc:1428] Setting key a4f79947195b3fe14204b3aa97a203fc
I20260925 10:13:43.103768 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Bootstrap starting.
I20260925 10:13:43.104527 11992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:43.104837 11992 log.cc:824] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:43.105575 11992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: No bootstrap required, opened a new log
I20260925 10:13:43.107887 11992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.108083 11992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:43.108139 11992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc48e05b61a4f00954f1103e8a8aa58, State: Initialized, Role: FOLLOWER
I20260925 10:13:43.108285 11992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.108352 11992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:43.108374 11992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:43.108436 11992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:43.109062 11992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.109189 11992 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc48e05b61a4f00954f1103e8a8aa58; no voters:
I20260925 10:13:43.109400 11992 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:43.109429 11997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:43.109625 11997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc48e05b61a4f00954f1103e8a8aa58, State: Running, Role: LEADER
I20260925 10:13:43.109808 11992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:43.109773 11997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.110128 11997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc48e05b61a4f00954f1103e8a8aa58. Latest consensus state: current_term: 1 leader_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } } }
I20260925 10:13:43.110193 11997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:43.110373 11999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } } }
I20260925 10:13:43.110525 11999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: This master's current role is: LEADER
W20260925 10:13:43.112133 12013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:13:43.112239 12013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:13:43.112308 12001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:43.112442 12001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:43.113950 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.102126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.3.152.254:32851)
0925 10:13:43.102396 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:43.102405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:43.102440 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:13:43.103007 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:43.103016 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:43.103082 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:43.103236 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:43.103870 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.105280 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.106148 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.106791 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.106908 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.109513 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:43.109517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:43.109517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:43.109542 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:43.112142 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.112566 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.112569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.112569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.112686 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.112882 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.112883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.112884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.113208 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:43.113356 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:43.113606 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:13:43.113752 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260925 10:13:43.114035 12001 catalog_manager.cc:1347] Generated new cluster ID: 44e5beb3633a46e08583a8b53cf2b950
I20260925 10:13:43.114086 12001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:43.119860 12001 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:43.120419 12001 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:43.123893 12001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Generated new TSK 0
I20260925 10:13:43.124087 12001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:13:43.150985 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:43.233718 12018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.233968 12018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.234009 12018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.235198 12018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.235263 12018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.235311 12018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:43.236884 12018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.237159 12018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.237411 12018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.243489 12024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.243618 12018 server_base.cc:1034] running on GCE node
W20260925 10:13:43.243497 12026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.243512 12023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.243944 12018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.244261 12018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.245409 12018 hybrid_clock.cc:648] HybridClock initialized: now 1790331223245397 us; error 28 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.248250 12018 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:13:43.248823 12018 webserver.cc:460] Webserver started at http://127.3.152.193:35451/ using document root <none> and password file <none>
I20260925 10:13:43.249109 12018 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.249161 12018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.249348 12018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:43.250730 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "a4fec0ec7d6c4ef9b4561bab860ff7fc"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "d4c31755b538102d2488ea24825da747"
server_key_iv: "d69abc2360c1d5472f0c699a16070e77"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.251120 12018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "a4fec0ec7d6c4ef9b4561bab860ff7fc"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "d4c31755b538102d2488ea24825da747"
server_key_iv: "d69abc2360c1d5472f0c699a16070e77"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.252532 12018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:13:43.253360 12033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.253536 12018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:13:43.253684 12018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "a4fec0ec7d6c4ef9b4561bab860ff7fc"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "d4c31755b538102d2488ea24825da747"
server_key_iv: "d69abc2360c1d5472f0c699a16070e77"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.253760 12018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:43.275816 12018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:43.277880 12018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:43.278043 12018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:43.278287 12018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:43.278599 12018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:43.278640 12018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.278680 12018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:43.278715 12018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.284886 12018 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:40029
I20260925 10:13:43.284961 12146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:40029 every 8 connection(s)
I20260925 10:13:43.285293 12018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:43.286432 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12018
I20260925 10:13:43.286551 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:43.286739 3683 external_mini_cluster.cc:1428] Setting key fee93d7f9f123a070ea2c00ea8778d6d
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:13:43.295929 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.286040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40899 (local address 127.3.152.254:32851)
0925 10:13:43.286101 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:43.286103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:43.286359 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:13:43.287064 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:43.287065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:43.287084 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:43.287144 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:43.288258 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.288765 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.290254 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.290472 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.290545 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.292856 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:43.292858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:43.292858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:43.292867 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:43.294320 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.294813 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.294815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.294816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.294880 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.295172 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.295174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.295174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.295345 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:43.295430 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:43.295741 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 10:13:43.295857 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:13:43.296062 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.286033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:40899)
0925 10:13:43.286225 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:43.286262 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:43.286944 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:43.287226 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:43.287231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:43.287898 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:43.288161 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.288166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.288863 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.288864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.290173 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.290176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.290589 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.290591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.290726 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.291322 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:43.291330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:43.292761 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:43.294440 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.294451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.294454 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.294717 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.294947 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.294948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.294949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.295069 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.295426 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:43.295432 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:43.295564 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 10:13:43.295684 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:13:43.295824 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260925 10:13:43.296849 12147 heartbeater.cc:347] Connected to a master server at 127.3.152.254:32851
I20260925 10:13:43.296963 12147 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:43.297160 12147 heartbeater.cc:511] Master 127.3.152.254:32851 requested a full tablet report, sending...
I20260925 10:13:43.297804 11956 ts_manager.cc:194] Registered new tserver with Master: a4fec0ec7d6c4ef9b4561bab860ff7fc (127.3.152.193:40029)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
I20260925 10:13:43.298722 11956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:40899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:13:43.316336 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:43.399507 12152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.399770 12152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.399812 12152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.401098 12152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.401188 12152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.401237 12152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:43.402823 12152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.403113 12152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.403362 12152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.408725 12157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.408982 12160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.408951 12152 server_base.cc:1034] running on GCE node
W20260925 10:13:43.409152 12158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.409644 12152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.410010 12152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.411186 12152 hybrid_clock.cc:648] HybridClock initialized: now 1790331223411150 us; error 52 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.415323 12152 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:13:43.415979 12152 webserver.cc:460] Webserver started at http://127.3.152.194:44619/ using document root <none> and password file <none>
I20260925 10:13:43.416333 12152 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.416414 12152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.416574 12152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:43.418227 12152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "848b3b33ee4d44a9935ef26d3c39fd18"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "259e0628add8354b46574dc5169dc17e"
server_key_iv: "b946d3a78303a7d6bfdc98d774e17a0d"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.418686 12152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "848b3b33ee4d44a9935ef26d3c39fd18"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "259e0628add8354b46574dc5169dc17e"
server_key_iv: "b946d3a78303a7d6bfdc98d774e17a0d"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.420178 12152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 10:13:43.420977 12167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.421178 12152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:43.421260 12152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "848b3b33ee4d44a9935ef26d3c39fd18"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "259e0628add8354b46574dc5169dc17e"
server_key_iv: "b946d3a78303a7d6bfdc98d774e17a0d"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.421339 12152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:43.455168 12152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:43.457798 12152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:43.458009 12152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:43.458323 12152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:43.458706 12152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:43.458762 12152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.458791 12152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:43.458806 12152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.465996 12152 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:43225
I20260925 10:13:43.466063 12280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:43225 every 8 connection(s)
I20260925 10:13:43.466440 12152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:43.471859 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12152
I20260925 10:13:43.472009 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:43.472226 3683 external_mini_cluster.cc:1428] Setting key 0fb42c0287f21f616c7d67ef3cb7eb54
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:43.477875 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.467249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:54855 (local address 127.3.152.254:32851)
0925 10:13:43.467319 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:43.467321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:43.467538 (+ 217us) server_negotiation.cc:407] Connection header received
0925 10:13:43.468109 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:43.468111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:43.468123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:43.468188 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:43.469172 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.469715 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.470848 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.471087 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.471161 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.473878 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:43.473879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:43.473879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:43.473887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:43.476408 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.476872 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.476874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.476874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.476934 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.477193 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.477194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.477195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.477324 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:43.477397 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:43.477686 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:13:43.477775 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:13:43.478114 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.467214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:54855)
0925 10:13:43.467393 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:43.467440 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:43.467953 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:43.468248 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:43.468251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:43.468854 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:43.469061 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.469066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.469828 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.469829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.470733 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.470735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.471192 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.471194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.471324 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.471936 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:43.471949 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:43.473771 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:43.476529 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.476538 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.476541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.476784 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.476999 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.476999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.477000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.477128 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.477390 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:43.477397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:43.477549 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 10:13:43.477666 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 10:13:43.477846 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260925 10:13:43.478653 12281 heartbeater.cc:347] Connected to a master server at 127.3.152.254:32851
I20260925 10:13:43.478793 12281 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:43.479001 12281 heartbeater.cc:511] Master 127.3.152.254:32851 requested a full tablet report, sending...
I20260925 10:13:43.479444 11956 ts_manager.cc:194] Registered new tserver with Master: 848b3b33ee4d44a9935ef26d3c39fd18 (127.3.152.194:43225)
I20260925 10:13:43.479861 11956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:54855
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:13:43.499135 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:43.585096 12286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.585325 12286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.585350 12286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.586570 12286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.586616 12286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.586644 12286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:43.588193 12286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.588465 12286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.588716 12286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.594394 12292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.594447 12294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.594393 12291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.594521 12286 server_base.cc:1034] running on GCE node
I20260925 10:13:43.594869 12286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.595103 12286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.596252 12286 hybrid_clock.cc:648] HybridClock initialized: now 1790331223596244 us; error 33 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.599076 12286 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:13:43.599570 12286 webserver.cc:460] Webserver started at http://127.3.152.195:42269/ using document root <none> and password file <none>
I20260925 10:13:43.599825 12286 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.599876 12286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.600062 12286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:43.601187 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "774e3e6a01704cc78bd2ad64eb9da2e7"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "a9b04d59e925ca893070aa4affb09221"
server_key_iv: "421c34e33bd1da283487692208458497"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.601557 12286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "774e3e6a01704cc78bd2ad64eb9da2e7"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "a9b04d59e925ca893070aa4affb09221"
server_key_iv: "421c34e33bd1da283487692208458497"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.602928 12286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:43.603899 12301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.604076 12286 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:43.604166 12286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "774e3e6a01704cc78bd2ad64eb9da2e7"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "a9b04d59e925ca893070aa4affb09221"
server_key_iv: "421c34e33bd1da283487692208458497"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.604228 12286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:43.616791 12286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:43.618688 12286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:43.618814 12286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:43.618999 12286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:43.619341 12286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:43.619375 12286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.619419 12286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:43.619446 12286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.625661 12286 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:40205
I20260925 10:13:43.625737 12414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:40205 every 8 connection(s)
I20260925 10:13:43.626097 12286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:13:43.634337 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12286
I20260925 10:13:43.634459 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:43.634640 3683 external_mini_cluster.cc:1428] Setting key 839a6773c30fe0a31a5a8060d59ab80b
I20260925 10:13:43.635457 11995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.626883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53509 (local address 127.3.152.254:32851)
0925 10:13:43.626966 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:43.626969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:43.627157 (+ 188us) server_negotiation.cc:407] Connection header received
0925 10:13:43.627640 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:43.627641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:43.627651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:43.627698 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:43.628756 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.629260 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.630265 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.630446 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.630499 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.632783 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:43.632785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:43.632785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:43.632792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:43.633995 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.634420 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.634421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.634422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.634490 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:43.634766 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:43.634767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:43.634768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:43.634900 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:43.634985 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:43.635282 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 10:13:43.635366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:13:43.635653 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:43.626851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:53509)
0925 10:13:43.627031 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:43.627069 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:43.627555 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:43.627770 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:43.627773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:43.628454 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:43.628667 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.628672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.629345 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.629345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.630191 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:43.630193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:43.630520 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:43.630520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:43.630623 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:43.631208 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:43.631217 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:43.632695 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:43.634067 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.634073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.634075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.634317 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.634565 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:13:43.634566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:13:43.634566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:13:43.634679 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:13:43.634977 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:43.634984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:43.635129 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:13:43.635255 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 10:13:43.635419 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260925 10:13:43.636114 12415 heartbeater.cc:347] Connected to a master server at 127.3.152.254:32851
I20260925 10:13:43.636231 12415 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:43.636408 12415 heartbeater.cc:511] Master 127.3.152.254:32851 requested a full tablet report, sending...
I20260925 10:13:43.636776 11956 ts_manager.cc:194] Registered new tserver with Master: 774e3e6a01704cc78bd2ad64eb9da2e7 (127.3.152.195:40205)
I20260925 10:13:43.637162 11956 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:53509
I20260925 10:13:43.646265 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
Entry for principal oryx/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab.
WARNING: no policy specified for oryx/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
Entry for principal oryx/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab.
WARNING: no policy specified for oryx/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
Entry for principal oryx/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab.
I20260925 10:13:43.718694 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12018
I20260925 10:13:43.723678 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12152
I20260925 10:13:43.728657 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12286
I20260925 10:13:43.732925 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 11924
I20260925 10:13:43.738399 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:32851
--webserver_interface=127.3.152.254
--webserver_port=46427
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:32851
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:13:43.822806 12424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.823064 12424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.823104 12424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.824414 12424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:43.824476 12424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.824499 12424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.824532 12424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:43.824556 12424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:43.826161 12424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:32851
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:32851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=46427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.826457 12424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.826684 12424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.831717 12429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.831712 12432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.831951 12424 server_base.cc:1034] running on GCE node
W20260925 10:13:43.831821 12430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.832340 12424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.832633 12424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.833828 12424 hybrid_clock.cc:648] HybridClock initialized: now 1790331223833806 us; error 32 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.836637 12424 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:13:43.837155 12424 webserver.cc:460] Webserver started at http://127.3.152.254:46427/ using document root <none> and password file <none>
I20260925 10:13:43.837415 12424 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.837509 12424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.838954 12424 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:13:43.839664 12439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.839854 12424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:43.840026 12424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0bc48e05b61a4f00954f1103e8a8aa58"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "8eddb36d337115cb682e9980bd8829d6"
server_key_iv: "85e6cf2e970a6acc1298a9b8ffe9f7b8"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.840495 12424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:43.852787 12424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:43.854925 12424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:43.855123 12424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:43.859391 12424 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:32851
I20260925 10:13:43.859426 12491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:32851 every 8 connection(s)
I20260925 10:13:43.859865 12424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:43.861518 12492 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:13:43.862159 12492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Bootstrap starting.
I20260925 10:13:43.863689 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12424
I20260925 10:13:43.864053 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:40029
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:32851
--webserver_port=35451
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:13:43.870949 12492 log.cc:824] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:43.872774 12492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:13:43.873082 12492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Bootstrap complete.
I20260925 10:13:43.875386 12492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.875648 12492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc48e05b61a4f00954f1103e8a8aa58, State: Initialized, Role: FOLLOWER
I20260925 10:13:43.875830 12492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.875900 12492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:43.875950 12492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:43.876041 12492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:13:43.877101 12492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.877225 12492 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc48e05b61a4f00954f1103e8a8aa58; no voters:
I20260925 10:13:43.877434 12492 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:13:43.877480 12496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:13:43.877609 12496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [term 2 LEADER]: Becoming Leader. State: Replica: 0bc48e05b61a4f00954f1103e8a8aa58, State: Running, Role: LEADER
I20260925 10:13:43.877696 12496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } }
I20260925 10:13:43.877731 12492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:43.877976 12496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc48e05b61a4f00954f1103e8a8aa58. Latest consensus state: current_term: 2 leader_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } } }
I20260925 10:13:43.878053 12496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:43.877981 12498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc48e05b61a4f00954f1103e8a8aa58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 32851 } } }
I20260925 10:13:43.878222 12498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:43.878731 12500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:43.879303 12500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:43.879482 12500 catalog_manager.cc:1259] Loaded cluster ID: 44e5beb3633a46e08583a8b53cf2b950
I20260925 10:13:43.879688 12500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:43.880581 12500 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:43.880769 12500 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0bc48e05b61a4f00954f1103e8a8aa58: Loaded TSK: 0
I20260925 10:13:43.880970 12500 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:13:43.972234 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:43.972483 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:43.972527 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:43.973765 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:43.973834 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:43.973881 12494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:43.975390 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.193.keytab
--principal=oryx/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:40029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=35451
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:43.975659 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:43.975903 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:43.981666 12518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.981771 12519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:43.981680 12521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:43.982131 12494 server_base.cc:1034] running on GCE node
I20260925 10:13:43.982367 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:43.982635 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:43.983830 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790331223983811 us; error 35 us; skew 500 ppm
Sep 25 10:13:43 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331223, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:43.986734 12494 init.cc:377] Logged in from keytab as oryx/127.3.152.193@KRBTEST.COM (short username oryx)
I20260925 10:13:43.987216 12494 webserver.cc:460] Webserver started at http://127.3.152.193:35451/ using document root <none> and password file <none>
I20260925 10:13:43.987449 12494 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:43.987531 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:43.989015 12494 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:43.989666 12528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:43.989890 12494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:13:43.989992 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "a4fec0ec7d6c4ef9b4561bab860ff7fc"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "d4c31755b538102d2488ea24825da747"
server_key_iv: "d69abc2360c1d5472f0c699a16070e77"
server_key_version: "encryptionkey@0"
I20260925 10:13:43.990477 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:44.021080 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:44.023736 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:44.023911 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:44.024233 12494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:44.024626 12494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:44.024678 12494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.024710 12494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:44.024736 12494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.032292 12494 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:40029
I20260925 10:13:44.032366 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:40029 every 8 connection(s)
I20260925 10:13:44.032771 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:44.040501 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12494
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.040817 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:43225
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:32851
--webserver_port=44619
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.042623 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.033690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:41141)
0925 10:13:44.033889 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.033940 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.034431 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.034921 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.034925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.035522 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.035722 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.035728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.037455 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.037456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.038286 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.038288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.039073 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.039073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.039171 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.039746 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.039778 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.042353 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
W20260925 10:13:44.042735 12645 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.033807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:41141 (local address 127.3.152.254:32851)
0925 10:13:44.034004 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.034014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.034060 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:13:44.034650 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.034659 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.034719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.034880 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.035865 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.037329 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.038371 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.038981 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.039096 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.042541 (+ 3445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:41141: BlockingRecv error: recv got EOF from 127.3.152.193:41141 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
W20260925 10:13:44.043303 12642 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:32851 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.152.254:32851: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.049474 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.043570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:41365)
0925 10:13:44.043619 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.043630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.043667 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.043808 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.043809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.043995 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.044065 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.044067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.045032 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.045033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.045905 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.045908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.046465 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.046466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.046559 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.047129 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.047134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.049396 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:13:44.050052 12645 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.043635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:41365 (local address 127.3.152.254:32851)
0925 10:13:44.043674 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.043676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.043688 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:44.043717 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.043718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.043728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.043794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.044173 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.044951 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.046012 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.046371 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.046453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.049952 (+ 3499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:41365: BlockingRecv error: recv got EOF from 127.3.152.193:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.056005 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.049902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:49385)
0925 10:13:44.050336 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.050348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.050406 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.050633 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.050635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.050819 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.050868 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.050870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.051815 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.051816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.052742 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.052745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.053229 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.053230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.053292 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.053776 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.053781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.055887 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415}
W20260925 10:13:44.056140 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.049956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:49385 (local address 127.3.152.254:32851)
0925 10:13:44.050125 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.050127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.050390 (+ 263us) server_negotiation.cc:407] Connection header received
0925 10:13:44.050474 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.050475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.050485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.050560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.050962 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.051723 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.052847 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.053126 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.053217 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.056061 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:49385: BlockingRecv error: recv got EOF from 127.3.152.193:49385 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260925 10:13:44.056452 12642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:13:44.129427 12646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:44.129693 12646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:44.129766 12646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:44.130980 12646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:44.131053 12646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:44.131111 12646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:44.132753 12646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.194.keytab
--principal=oryx/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:43225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=44619
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:44.133039 12646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:44.133280 12646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:44.138772 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:44.138772 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:44.138777 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:44.138939 12646 server_base.cc:1034] running on GCE node
I20260925 10:13:44.139222 12646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:44.139497 12646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:44.140720 12646 hybrid_clock.cc:648] HybridClock initialized: now 1790331224140704 us; error 25 us; skew 500 ppm
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331224, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:44.143358 12646 init.cc:377] Logged in from keytab as oryx/127.3.152.194@KRBTEST.COM (short username oryx)
I20260925 10:13:44.143873 12646 webserver.cc:460] Webserver started at http://127.3.152.194:44619/ using document root <none> and password file <none>
I20260925 10:13:44.144171 12646 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:44.144253 12646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:44.145659 12646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:44.146349 12662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.146524 12646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:44.146687 12646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "848b3b33ee4d44a9935ef26d3c39fd18"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "259e0628add8354b46574dc5169dc17e"
server_key_iv: "b946d3a78303a7d6bfdc98d774e17a0d"
server_key_version: "encryptionkey@0"
I20260925 10:13:44.147076 12646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:44.173452 12646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:44.176060 12646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:44.176232 12646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:44.176484 12646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:44.176823 12646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:44.176877 12646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.176921 12646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:44.176946 12646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.183002 12646 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:43225
I20260925 10:13:44.183084 12775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:43225 every 8 connection(s)
I20260925 10:13:44.183439 12646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:44.187352 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12646
I20260925 10:13:44.187659 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:40205
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:32851
--webserver_port=42269
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:13:44.191558 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.184230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:46449 (local address 127.3.152.254:32851)
0925 10:13:44.184314 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.184317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.184549 (+ 232us) server_negotiation.cc:407] Connection header received
0925 10:13:44.185027 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.185028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.185041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.185092 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.186186 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.186683 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.187927 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.188216 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.188308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.191495 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:46449: BlockingRecv error: recv got EOF from 127.3.152.194:46449 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:13:44.191705 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.184234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:46449)
0925 10:13:44.184425 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.184466 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.184944 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.185170 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.185174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.185864 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.186080 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.186094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.186793 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.186794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.187846 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.187849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.188323 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.188324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.188464 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.189029 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.189038 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.191407 (+ 2369us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
W20260925 10:13:44.192157 12776 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:32851 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.152.254:32851: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.197747 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.192421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:45561)
0925 10:13:44.192539 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.192550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.192600 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.192744 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.192745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.192862 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.192914 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.192915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.193906 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.193907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.194486 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.194487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.195040 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.195041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.195107 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.195580 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.195585 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.197694 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260925 10:13:44.197897 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.192491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:45561 (local address 127.3.152.254:32851)
0925 10:13:44.192553 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.192556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.192591 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:13:44.192650 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.192651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.192666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.192745 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.192982 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.193802 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.194572 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.194967 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.195048 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.197813 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:45561: BlockingRecv error: recv got EOF from 127.3.152.194:45561 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.202466 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.198243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:54007)
0925 10:13:44.198290 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.198300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.198337 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.198485 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.198485 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.198614 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.198643 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.198644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.199271 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.199272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.199764 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.199765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.200179 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.200180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.200232 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.200669 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.200674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.202394 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 10:13:44.202665 12776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:13:44.202711 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.198319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:54007 (local address 127.3.152.254:32851)
0925 10:13:44.198376 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.198379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.198386 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:13:44.198406 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.198407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.198418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.198483 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.198719 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.199194 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.199842 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.200104 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.200171 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.202488 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:54007: BlockingRecv error: recv got EOF from 127.3.152.194:54007 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 10:13:44.273097 12779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:44.273358 12779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:44.273398 12779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:44.274601 12779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:44.274670 12779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:44.274715 12779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:44.276275 12779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/krb5kdc/oryx_127.3.152.195.keytab
--principal=oryx/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:40205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=42269
--tserver_master_addrs=127.3.152.254:32851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:44.276540 12779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:44.276782 12779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:13:44.282757 12779 server_base.cc:1034] running on GCE node
W20260925 10:13:44.282846 12784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:44.282816 12787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:44.282915 12785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:44.283192 12779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:44.283481 12779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:44.284672 12779 hybrid_clock.cc:648] HybridClock initialized: now 1790331224284654 us; error 26 us; skew 500 ppm
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331224, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:13:44.287333 12779 init.cc:377] Logged in from keytab as oryx/127.3.152.195@KRBTEST.COM (short username oryx)
I20260925 10:13:44.287799 12779 webserver.cc:460] Webserver started at http://127.3.152.195:42269/ using document root <none> and password file <none>
I20260925 10:13:44.288118 12779 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:44.288203 12779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:44.289628 12779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:44.290479 12794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.290783 12779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:44.290906 12779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "774e3e6a01704cc78bd2ad64eb9da2e7"
format_stamp: "Formatted at 2026-09-25 10:13:43 on dist-test-slave-dzwq"
server_key: "a9b04d59e925ca893070aa4affb09221"
server_key_iv: "421c34e33bd1da283487692208458497"
server_key_version: "encryptionkey@0"
I20260925 10:13:44.291325 12779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:44.308238 12779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:44.311108 12779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:44.311446 12779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:44.311842 12779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:44.312362 12779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:44.312431 12779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.312474 12779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:44.312521 12779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:44.318768 12779 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:40205
I20260925 10:13:44.318810 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:40205 every 8 connection(s)
I20260925 10:13:44.319212 12779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:44.323407 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12779
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 10:13:44.327698 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.320155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:60473 (local address 127.3.152.254:32851)
0925 10:13:44.320232 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.320234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.320490 (+ 256us) server_negotiation.cc:407] Connection header received
0925 10:13:44.321046 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.321047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.321058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.321117 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.322119 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.322673 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.323798 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.324138 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.324222 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.327638 (+ 3416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:60473: BlockingRecv error: recv got EOF from 127.3.152.195:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:13:44.327787 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.320167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:60473)
0925 10:13:44.320355 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.320393 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.320938 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.321194 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.321198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.321804 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.322019 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.322024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.322763 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.322764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.323652 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.323655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.324244 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.324246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.324377 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.325006 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.325016 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.327567 (+ 2551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
W20260925 10:13:44.328079 12908 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:32851 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.152.254:32851: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.332492 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.324067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.3.152.254:32851)
0925 10:13:44.324121 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.324123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.324133 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:44.324203 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.324205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.324219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.324293 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.325117 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.325650 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.326670 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.326955 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.327029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.328526 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:44.328530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:44.328531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:13:44.328542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:44.330661 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:44.331124 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:44.331127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:44.331127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:44.331237 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:13:44.331512 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:13:44.331513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:13:44.331513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:13:44.331925 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:44.332115 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:44.332262 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:13:44.332393 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.332984 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.328359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:42025)
0925 10:13:44.328398 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.328416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.328476 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.328633 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.328634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.328764 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.328808 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.328809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.329436 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.329437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.329935 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.329937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.330303 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.330304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.330380 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.330834 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.330839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.332904 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 10:13:44.333029 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.328414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:42025 (local address 127.3.152.254:32851)
0925 10:13:44.328479 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.328480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.328489 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:44.328535 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.328536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.328546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.328616 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.328898 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.329357 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.330033 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.330232 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.330305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.332968 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:42025: BlockingRecv error: recv got EOF from 127.3.152.195:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:44 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:44.337841 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:44.333279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:58585)
0925 10:13:44.333332 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:44.333344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:44.333386 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:44.333521 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:44.333521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:44.333680 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:44.333716 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.333718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.334392 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.334393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.335136 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:44.335138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.335463 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.335464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:44.335502 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.335999 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:44.336004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:44.337793 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:13:44.337996 12908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:13:44.338035 12647 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:44.333360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:58585 (local address 127.3.152.254:32851)
0925 10:13:44.333402 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:44.333404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:44.333412 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:13:44.333440 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:44.333441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:44.333448 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:44.333514 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:44.333820 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.334302 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.335216 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:44.335396 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:44.335451 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:44.337977 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:58585: BlockingRecv error: recv got EOF from 127.3.152.195:58585 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:45.061933 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:45.056923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:60593)
0925 10:13:45.057108 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:45.057122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:45.057189 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:45.057356 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:45.057357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:45.057524 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:45.057577 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.057578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.058342 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.058343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.058910 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.058912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.059302 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.059302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.059341 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.059795 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:45.059800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:45.061859 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
W20260925 10:13:45.062140 12913 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:45.056999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:60593 (local address 127.3.152.254:32851)
0925 10:13:45.057160 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:45.057162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:45.057174 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:45.057268 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:45.057269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:45.057289 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:45.057350 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:45.057666 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.058245 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.058998 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.059223 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.059289 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.062036 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:60593: BlockingRecv error: recv got EOF from 127.3.152.193:60593 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:45.208454 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:45.203165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:46779)
0925 10:13:45.203365 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:45.203380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:45.203484 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:45.203674 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:45.203675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:45.203865 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:45.203913 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.203915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.204716 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.204717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.205297 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.205299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.205699 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.205700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.205758 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.206293 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:45.206301 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:45.208340 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":105,"threads_started":1}
W20260925 10:13:45.208640 12913 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:45.203302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:46779 (local address 127.3.152.254:32851)
0925 10:13:45.203391 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:45.203393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:45.203461 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:13:45.203572 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:45.203573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:45.203593 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:45.203665 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:45.204048 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.204551 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.205397 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.205601 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.205698 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.208488 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:46779: BlockingRecv error: recv got EOF from 127.3.152.194:46779 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:45 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:45.343999 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:45.338496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:34629)
0925 10:13:45.338701 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:45.338716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:45.338787 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:45.338925 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:45.338926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:45.339086 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:45.339136 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.339138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.339884 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.339885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.340532 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:45.340535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.340947 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.340948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:45.341001 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.341575 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:45.341580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:45.343863 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
W20260925 10:13:45.344149 12913 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:45.338632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:34629 (local address 127.3.152.254:32851)
0925 10:13:45.338715 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:45.338717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:45.338782 (+ 65us) server_negotiation.cc:407] Connection header received
0925 10:13:45.338835 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:45.338836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:45.338849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:45.338903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:45.339235 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.339745 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.340638 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:45.340852 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:45.340914 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:45.344066 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:34629: BlockingRecv error: recv got EOF from 127.3.152.195:34629 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:46.067721 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:46.062418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:40967)
0925 10:13:46.062588 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:46.062599 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:46.062656 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:46.062858 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:46.062859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:46.063027 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:46.063076 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.063077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.063793 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.063794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.064349 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.064351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.064745 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.064746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.064786 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.065297 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:46.065301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:46.067569 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
W20260925 10:13:46.067735 12917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:46.062541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40967 (local address 127.3.152.254:32851)
0925 10:13:46.062736 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:46.062738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:46.062747 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:46.062768 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:46.062769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:46.062778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:46.062839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:46.063179 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.063717 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.064432 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.064644 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.064718 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.067672 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:40967: BlockingRecv error: recv got EOF from 127.3.152.193:40967 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":131,"threads_started":1}
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:46.214022 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:46.208996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:60653)
0925 10:13:46.209133 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:46.209152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:46.209209 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:46.209332 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:46.209332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:46.209516 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:46.209565 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.209567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.210336 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.210337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.210871 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.210873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.211283 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.211284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.211342 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.211942 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:46.211947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:46.213922 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260925 10:13:46.214095 12917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:46.209095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:60653 (local address 127.3.152.254:32851)
0925 10:13:46.209194 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:46.209196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:46.209207 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:46.209259 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:46.209260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:46.209273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:46.209327 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:46.209657 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.210204 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.210967 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.211195 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.211285 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.214010 (+ 2725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:60653: BlockingRecv error: recv got EOF from 127.3.152.194:60653 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:46 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:46.350069 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:46.344497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:47453)
0925 10:13:46.344671 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:46.344686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:46.344775 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:46.345035 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:46.345036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:46.345246 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:46.345300 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.345302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.346126 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.346128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.346755 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:46.346758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.347326 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.347328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:46.347396 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.348011 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:46.348016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:46.350010 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
W20260925 10:13:46.350271 12917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:46.344647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:47453 (local address 127.3.152.254:32851)
0925 10:13:46.344731 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:46.344733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:46.344750 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:13:46.344909 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:46.344910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:46.344935 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:46.345025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:46.345404 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.346006 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.346864 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:46.347162 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:46.347252 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:46.350211 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:47453: BlockingRecv error: recv got EOF from 127.3.152.195:47453 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:47.073683 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:47.068392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:35827)
0925 10:13:47.068544 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:47.068565 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:47.068629 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:47.068915 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:47.068916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:47.069142 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:47.069205 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.069207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.070006 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.070007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.070562 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.070570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.070968 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.070969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.071045 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.071668 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:47.071673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:47.073528 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
W20260925 10:13:47.073686 12921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:47.068471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:35827 (local address 127.3.152.254:32851)
0925 10:13:47.068611 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:47.068612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:47.068641 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:13:47.068774 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:47.068775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:47.068790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:47.068875 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:47.069288 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.069914 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.070664 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.070882 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.070956 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.073621 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:35827: BlockingRecv error: recv got EOF from 127.3.152.193:35827 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:47.219929 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:47.214725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:52617)
0925 10:13:47.214935 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:47.214948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:47.215040 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:47.215195 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:47.215196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:47.215397 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:47.215445 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.215447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.216219 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.216220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.216764 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.216766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.217151 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.217152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.217218 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.217748 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:47.217752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:47.219839 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":128,"threads_started":1}
W20260925 10:13:47.220041 12921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:47.214855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:52617 (local address 127.3.152.254:32851)
0925 10:13:47.214935 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:47.214938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:47.215026 (+ 88us) server_negotiation.cc:407] Connection header received
0925 10:13:47.215100 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:47.215100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:47.215113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:47.215187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:47.215546 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.216125 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.216855 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.217061 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.217150 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.219950 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:52617: BlockingRecv error: recv got EOF from 127.3.152.194:52617 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:47 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:47.355144 12923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:47.350528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:48465)
0925 10:13:47.350716 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:47.350730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:47.350818 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:47.350951 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:47.350952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:47.351120 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:47.351170 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.351171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.351815 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.351816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.352369 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:47.352370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.352709 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.352709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:47.352753 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.353191 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:47.353196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:47.355083 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":62,"threads_started":1}
W20260925 10:13:47.355273 12921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:47.350753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:48465 (local address 127.3.152.254:32851)
0925 10:13:47.350806 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:47.350808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:47.350818 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:47.350869 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:47.350870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:47.350882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:47.350931 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:47.351246 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.351744 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.352445 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:47.352633 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:47.352690 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:47.355182 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:48465: BlockingRecv error: recv got EOF from 127.3.152.195:48465 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:48.084338 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:48.074263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:35931)
0925 10:13:48.074432 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:48.074447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:48.074523 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:48.074844 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:48.074845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:48.075075 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:48.075142 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.075145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.076292 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.076293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.077247 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.077250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.077819 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.077820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.077885 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.081942 (+ 4057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:48.081957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:48.084234 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260925 10:13:48.084435 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:48.074476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:35931 (local address 127.3.152.254:32851)
0925 10:13:48.074634 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:48.074636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:48.074647 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:48.074680 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:48.074681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:48.074697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:48.074775 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:48.075248 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.076200 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.077360 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.077731 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.077814 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.084355 (+ 6541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:35931: BlockingRecv error: recv got EOF from 127.3.152.193:35931 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:48.225203 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:48.220566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:41863)
0925 10:13:48.220685 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:48.220704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:48.220763 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:48.220922 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:48.220923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:48.221084 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:48.221133 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.221134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.221846 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.221847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.222378 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.222379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.222758 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.222758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.222798 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.223240 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:48.223245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:48.225143 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260925 10:13:48.225327 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:48.220633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:41863 (local address 127.3.152.254:32851)
0925 10:13:48.220708 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:48.220710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:48.220751 (+ 41us) server_negotiation.cc:407] Connection header received
0925 10:13:48.220828 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:48.220829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:48.220845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:48.220888 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:48.221202 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.221767 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.222453 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.222689 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.222746 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.225258 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:41863: BlockingRecv error: recv got EOF from 127.3.152.194:41863 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:48 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:48.360771 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:48.355733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:47909)
0925 10:13:48.355852 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:48.355871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:48.355925 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:48.356121 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:48.356122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:48.356295 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:48.356344 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.356346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.357037 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.357038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.357659 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:48.357662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.358055 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.358056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:48.358116 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.358690 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:48.358696 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:48.360690 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
W20260925 10:13:48.360875 12925 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:48.355812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:47909 (local address 127.3.152.254:32851)
0925 10:13:48.355914 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:48.355916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:48.355931 (+ 15us) server_negotiation.cc:407] Connection header received
0925 10:13:48.355997 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:48.355998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:48.356016 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:48.356107 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:48.356423 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.356955 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.357779 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:48.357979 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:48.358054 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:48.360810 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:47909: BlockingRecv error: recv got EOF from 127.3.152.195:47909 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:49.090233 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:49.084995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:41007)
0925 10:13:49.085215 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:49.085228 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:49.085295 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:49.085453 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:49.085454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:49.085676 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:49.085723 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.085725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.086501 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.086502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.087090 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.087092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.087479 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.087480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.087535 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.088253 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:49.088258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:49.090170 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":124,"threads_started":1}
W20260925 10:13:49.090397 12929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:49.085163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:41007 (local address 127.3.152.254:32851)
0925 10:13:49.085315 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:49.085316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:49.085326 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:49.085352 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:49.085352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:49.085363 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:49.085435 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:49.085828 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.086409 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.087186 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.087391 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.087452 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.090292 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:41007: BlockingRecv error: recv got EOF from 127.3.152.193:41007 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:49.230732 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:49.225778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:54695)
0925 10:13:49.225917 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:49.225936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:49.226017 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:49.226205 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:49.226205 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:49.226387 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:49.226433 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.226434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.227153 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.227154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.227695 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.227696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.228128 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.228129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.228192 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.228692 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:49.228698 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:49.230629 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260925 10:13:49.230777 12929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:49.225925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:54695 (local address 127.3.152.254:32851)
0925 10:13:49.225996 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:49.225998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:49.226008 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:49.226125 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:49.226126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:49.226138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:49.226197 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:49.226520 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.227065 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.227779 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.228034 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.228126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.230715 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:54695: BlockingRecv error: recv got EOF from 127.3.152.194:54695 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:49 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:49.366360 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:49.361308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:51647)
0925 10:13:49.361472 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:49.361484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:49.361536 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:49.361733 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:49.361734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:49.361904 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:49.361963 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.361965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.362690 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.362691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.363288 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:49.363289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.363674 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.363675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:49.363723 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.364243 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:49.364248 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:49.366254 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
W20260925 10:13:49.366390 12929 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:49.361426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:51647 (local address 127.3.152.254:32851)
0925 10:13:49.361506 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:49.361508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:49.361520 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:13:49.361585 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:49.361586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:49.361598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:49.361661 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:49.362062 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.362563 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.363367 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:49.363547 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:49.363656 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:49.366333 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:51647: BlockingRecv error: recv got EOF from 127.3.152.195:51647 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:50.095817 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:50.090844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:54037)
0925 10:13:50.091004 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:50.091015 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:50.091079 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:50.091233 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:50.091234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:50.091408 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:50.091457 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.091459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.092194 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.092196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.092788 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.092791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.093136 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.093137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.093180 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.093702 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:50.093706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:50.095746 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
W20260925 10:13:50.096026 12933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:50.090961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:54037 (local address 127.3.152.254:32851)
0925 10:13:50.091097 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:50.091099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:50.091135 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:13:50.091161 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:50.091162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:50.091172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:50.091221 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:50.091525 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.092110 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.092870 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.093067 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.093124 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.095887 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:54037: BlockingRecv error: recv got EOF from 127.3.152.193:54037 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:50.237112 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:50.231401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:39519)
0925 10:13:50.231561 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:50.231580 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:50.231647 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:50.231796 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:50.231797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:50.232016 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:50.232107 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.232110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.233227 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.233228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.234086 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.234088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.234476 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.234477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.234532 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.235029 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:50.235033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:50.237044 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260925 10:13:50.237317 12933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:50.231504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:39519 (local address 127.3.152.254:32851)
0925 10:13:50.231636 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:50.231638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:50.231651 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:13:50.231699 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:50.231700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:50.231711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:50.231775 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:50.232217 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.233131 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.234173 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.234392 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.234476 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.237225 (+ 2749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:39519: BlockingRecv error: recv got EOF from 127.3.152.194:39519 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:50 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:50.372424 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:50.367049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:55455)
0925 10:13:50.367213 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:50.367228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:50.367288 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:50.367428 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:50.367428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:50.367631 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:50.367693 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.367696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.368688 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.368689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.369284 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:50.369286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.369652 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.369654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:50.369707 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.370193 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:50.370197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:50.372363 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
W20260925 10:13:50.372604 12933 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:50.367141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:55455 (local address 127.3.152.254:32851)
0925 10:13:50.367229 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:50.367231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:50.367286 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:13:50.367334 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:50.367335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:50.367348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:50.367409 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:50.367786 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.368598 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.369364 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:50.369545 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:50.369614 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:50.372504 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:55455: BlockingRecv error: recv got EOF from 127.3.152.195:55455 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:51.101356 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:51.096298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:40107)
0925 10:13:51.096441 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:51.096452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:51.096504 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:51.096833 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:51.096835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:51.097036 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:51.097118 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.097120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.097877 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.097878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.098426 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.098428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.098866 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.098867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.098915 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.099486 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:51.099491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:51.101293 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260925 10:13:51.101490 12937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:51.096412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40107 (local address 127.3.152.254:32851)
0925 10:13:51.096545 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:51.096547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:51.096555 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:13:51.096590 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:51.096590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:51.096606 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:51.096733 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:51.097217 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.097794 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.098528 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.098790 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.098854 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.101397 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:40107: BlockingRecv error: recv got EOF from 127.3.152.193:40107 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:51.242496 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:51.237647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:49735)
0925 10:13:51.237782 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:51.237795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:51.237892 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:51.238066 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:51.238067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:51.238248 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:51.238295 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.238298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.238993 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.238995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.239524 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.239526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.239921 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.239921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.240022 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.240554 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:51.240559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:51.242393 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260925 10:13:51.242553 12937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:51.237714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:49735 (local address 127.3.152.254:32851)
0925 10:13:51.237774 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:51.237776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:51.237894 (+ 118us) server_negotiation.cc:407] Connection header received
0925 10:13:51.237958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:51.237960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:51.237974 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:51.238043 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:51.238383 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.238903 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.239619 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.239827 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.239918 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.242490 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:49735: BlockingRecv error: recv got EOF from 127.3.152.194:49735 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:51 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:51.378141 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:51.373007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:56927)
0925 10:13:51.373204 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:51.373214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:51.373272 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:51.373500 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:51.373502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:51.373684 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:51.373738 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.373739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.374445 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.374447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.375010 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:51.375013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.375400 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.375401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:51.375458 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.376109 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:51.376114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:51.378077 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":126,"threads_started":1}
W20260925 10:13:51.378273 12937 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:51.373066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:56927 (local address 127.3.152.254:32851)
0925 10:13:51.373133 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:51.373135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:51.373308 (+ 173us) server_negotiation.cc:407] Connection header received
0925 10:13:51.373352 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:51.373353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:51.373368 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:51.373436 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:51.373836 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.374347 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.375119 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:51.375316 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:51.375386 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:51.378189 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:56927: BlockingRecv error: recv got EOF from 127.3.152.195:56927 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:52.107250 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:52.101965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:47131)
0925 10:13:52.102144 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:52.102157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:52.102229 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:52.102545 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:52.102546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:52.102749 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:52.102819 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.102821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.103702 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.103703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.104327 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.104329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.104744 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.104745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.104801 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.105362 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:52.105368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:52.107181 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
W20260925 10:13:52.107479 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:52.102185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:47131 (local address 127.3.152.254:32851)
0925 10:13:52.102361 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:52.102363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:52.102372 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:52.102397 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:52.102398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:52.102410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:52.102485 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:52.102930 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.103580 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.104412 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.104603 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.104713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.107393 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:47131: BlockingRecv error: recv got EOF from 127.3.152.193:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:52.248742 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:52.243151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:34495)
0925 10:13:52.243436 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:52.243455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:52.243526 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:52.243709 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:52.243710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:52.243883 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:52.243934 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.243935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.244964 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.244965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.245520 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.245522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.246200 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.246201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.246269 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.246850 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:52.246855 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:52.248684 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
W20260925 10:13:52.248919 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:52.243243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:34495 (local address 127.3.152.254:32851)
0925 10:13:52.243462 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:52.243464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:52.243509 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:13:52.243582 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:52.243583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:52.243595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:52.243659 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:52.244031 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.244872 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.245635 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.245982 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.246055 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.248812 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:34495: BlockingRecv error: recv got EOF from 127.3.152.194:34495 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:52 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:52.383829 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:52.378655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:48105)
0925 10:13:52.378838 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:52.378849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:52.378924 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:52.379060 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:52.379061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:52.379280 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:52.379335 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.379338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.380185 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.380186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.380753 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:52.380755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.381139 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.381140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:52.381181 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.381743 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:52.381748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:52.383755 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
W20260925 10:13:52.383975 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:52.378747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:48105 (local address 127.3.152.254:32851)
0925 10:13:52.378804 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:52.378807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:52.378922 (+ 115us) server_negotiation.cc:407] Connection header received
0925 10:13:52.378972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:52.378972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:52.378986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:52.379039 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:52.379421 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.380104 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.380848 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:52.381065 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:52.381128 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:52.383879 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:48105: BlockingRecv error: recv got EOF from 127.3.152.195:48105 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:53.117177 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:53.107855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:45901)
0925 10:13:53.108064 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:53.108077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:53.108157 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:53.108305 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:53.108307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:53.108514 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:53.108565 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.108567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.109336 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.109337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.109884 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.109887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.110275 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.110276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.110322 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.114600 (+ 4278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:53.114608 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:53.117116 (+ 2508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
W20260925 10:13:53.117304 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:53.107991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:45901 (local address 127.3.152.254:32851)
0925 10:13:53.108146 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:53.108148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:53.108157 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:53.108210 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:53.108210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:53.108221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:53.108282 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:53.108706 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.109238 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.109984 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.110199 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.110258 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.117217 (+ 6959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:45901: BlockingRecv error: recv got EOF from 127.3.152.193:45901 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:53.254835 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:53.249378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:51001)
0925 10:13:53.249592 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:53.249604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:53.249676 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:53.249884 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:53.249885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:53.250104 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:53.250161 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.250163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.251053 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.251054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.251612 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.251614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.252056 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.252057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.252118 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.252669 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:53.252674 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:53.254760 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260925 10:13:53.254985 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:53.249467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:51001 (local address 127.3.152.254:32851)
0925 10:13:53.249541 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:53.249544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:53.249654 (+ 110us) server_negotiation.cc:407] Connection header received
0925 10:13:53.249748 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:53.249749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:53.249784 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:53.249851 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:53.250287 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.250957 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.251699 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.251917 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.252051 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.254891 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:51001: BlockingRecv error: recv got EOF from 127.3.152.194:51001 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:53 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:53.389323 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:53.384369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:53643)
0925 10:13:53.384634 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:53.384645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:53.384713 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:53.384857 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:53.384857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:53.385045 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:53.385096 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.385097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.385802 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.385803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.386347 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:53.386349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.386725 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.386726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:53.386766 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.387230 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:53.387235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:53.389258 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
W20260925 10:13:53.389434 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:53.384505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53643 (local address 127.3.152.254:32851)
0925 10:13:53.384567 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:53.384570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:53.384715 (+ 145us) server_negotiation.cc:407] Connection header received
0925 10:13:53.384769 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:53.384770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:53.384784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:53.384837 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:53.385159 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.385715 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.386434 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:53.386652 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:53.386717 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:53.389367 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:53643: BlockingRecv error: recv got EOF from 127.3.152.195:53643 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:54.122927 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:54.117731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:40861)
0925 10:13:54.117911 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:54.117923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:54.117977 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:54.118132 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:54.118133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:54.118309 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:54.118358 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.118359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.119098 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.119099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.119646 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.119648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.120117 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.120118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.120182 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.120698 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:54.120702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:54.122835 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
W20260925 10:13:54.123080 12949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:54.117833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40861 (local address 127.3.152.254:32851)
0925 10:13:54.117978 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:54.117980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:54.118012 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:13:54.118034 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:54.118034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:54.118045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:54.118118 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:54.118436 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.119006 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.119746 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.120028 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.120117 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.122958 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:40861: BlockingRecv error: recv got EOF from 127.3.152.193:40861 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:54.260504 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:54.255386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:39219)
0925 10:13:54.255542 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:54.255552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:54.255601 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:54.255776 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:54.255776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:54.255948 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:54.256058 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.256060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.256880 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.256881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.257428 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.257429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.257892 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.257894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.257945 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.258508 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:54.258513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:54.260439 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260925 10:13:54.260704 12949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:54.255508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:39219 (local address 127.3.152.254:32851)
0925 10:13:54.255632 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:54.255634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:54.255643 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:13:54.255672 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:54.255673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:54.255685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:54.255762 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:54.256144 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.256793 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.257527 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.257799 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.257867 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.260561 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:39219: BlockingRecv error: recv got EOF from 127.3.152.194:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:54 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:54.395341 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:54.390021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:34465)
0925 10:13:54.390195 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:54.390213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:54.390300 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:54.390459 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:54.390460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:54.390662 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:54.390714 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.390716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.391487 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.391488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.392122 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:54.392124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.392549 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.392550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:54.392623 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.393085 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:54.393089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:54.395240 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
W20260925 10:13:54.395475 12949 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:54.390234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:34465 (local address 127.3.152.254:32851)
0925 10:13:54.390307 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:54.390309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:54.390319 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:13:54.390371 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:54.390372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:54.390388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:54.390441 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:54.390815 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.391386 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.392219 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:54.392450 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:54.392544 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:54.395349 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:34465: BlockingRecv error: recv got EOF from 127.3.152.195:34465 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:55.128950 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:55.123596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:49571)
0925 10:13:55.123807 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:55.123817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:55.123875 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:55.124095 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:55.124096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:55.124278 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:55.124326 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.124327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.125331 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.125332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.125966 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.125968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.126331 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.126332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.126372 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.126901 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:55.126906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:55.128880 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":137,"threads_started":1}
W20260925 10:13:55.129133 12953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:55.123777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:49571 (local address 127.3.152.254:32851)
0925 10:13:55.123921 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:55.123923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:55.123936 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:13:55.123982 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:55.123983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:55.123994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:55.124069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:55.124409 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.125231 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.126054 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.126260 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.126318 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.129014 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:49571: BlockingRecv error: recv got EOF from 127.3.152.193:49571 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:55.266089 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:55.261144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:53219)
0925 10:13:55.261336 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:55.261348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:55.261424 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:55.261590 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:55.261591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:55.261802 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:55.261852 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.261854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.262591 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.262592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.263238 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.263241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.263618 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.263620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.263684 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.264179 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:55.264184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:55.266018 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
W20260925 10:13:55.266295 12953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:55.261268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53219 (local address 127.3.152.254:32851)
0925 10:13:55.261349 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:55.261351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:55.261453 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:13:55.261486 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:55.261486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:55.261498 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:55.261555 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:55.261952 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.262490 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.263332 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.263527 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.263600 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.266166 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:53219: BlockingRecv error: recv got EOF from 127.3.152.194:53219 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:55 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:55.400946 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:55.396014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:57131)
0925 10:13:55.396164 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:55.396176 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:55.396262 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:55.396407 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:55.396408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:55.396597 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:55.396691 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.396693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.397420 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.397422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.397979 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:55.397980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.398379 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.398380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:55.398438 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.398964 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:55.398977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:55.400886 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
W20260925 10:13:55.401202 12953 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:55.396075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:57131 (local address 127.3.152.254:32851)
0925 10:13:55.396165 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:55.396167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:55.396233 (+ 66us) server_negotiation.cc:407] Connection header received
0925 10:13:55.396328 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:55.396329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:55.396344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:55.396392 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:55.396785 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.397328 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.398073 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:55.398297 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:55.398380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:55.401036 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:57131: BlockingRecv error: recv got EOF from 127.3.152.195:57131 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:56.134339 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:56.129534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:46909)
0925 10:13:56.129749 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:56.129760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:56.129818 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:56.129959 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:56.129960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:56.130177 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:56.130224 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.130226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.130907 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.130908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.131464 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.131466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.131812 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.131813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.131858 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.132368 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:56.132372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:56.134278 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
W20260925 10:13:56.134477 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:56.129630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:46909 (local address 127.3.152.254:32851)
0925 10:13:56.129779 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:56.129781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:56.129811 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:13:56.129872 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:56.129873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:56.129883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:56.129945 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:56.130308 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.130827 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.131544 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.131739 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.131797 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.134384 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:46909: BlockingRecv error: recv got EOF from 127.3.152.193:46909 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":109,"threads_started":1}
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:56.271819 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:56.266631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:33133)
0925 10:13:56.266867 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:56.266882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:56.266978 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:56.267186 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:56.267187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:56.267381 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:56.267428 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.267429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.268230 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.268231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.268778 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.268781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.269182 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.269183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.269250 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.269770 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:56.269775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:56.271717 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":128,"threads_started":1}
W20260925 10:13:56.271872 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:56.266794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:33133 (local address 127.3.152.254:32851)
0925 10:13:56.266871 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:56.266873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:56.266957 (+ 84us) server_negotiation.cc:407] Connection header received
0925 10:13:56.267048 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:56.267049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:56.267066 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:56.267172 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:56.267517 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.268142 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.268870 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.269082 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.269182 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.271806 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:33133: BlockingRecv error: recv got EOF from 127.3.152.194:33133 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:56 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:56.406836 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:56.401529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:58653)
0925 10:13:56.401712 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:56.401731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:56.401813 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:56.401956 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:56.401956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:56.402135 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:56.402182 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.402184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.402869 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.402870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.403416 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:56.403418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.403767 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.403768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:56.403827 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.404320 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:56.404327 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:56.406771 (+ 2444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
W20260925 10:13:56.406960 12957 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:56.401663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:58653 (local address 127.3.152.254:32851)
0925 10:13:56.401732 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:56.401734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:56.401808 (+ 74us) server_negotiation.cc:407] Connection header received
0925 10:13:56.401869 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:56.401870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:56.401883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:56.401936 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:56.402248 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.402772 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.403483 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:56.403673 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:56.403736 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:56.406878 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:58653: BlockingRecv error: recv got EOF from 127.3.152.195:58653 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:57.140368 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:57.134959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:51713)
0925 10:13:57.135139 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:57.135154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:57.135227 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:57.135373 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:57.135373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:57.135549 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:57.135595 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.135596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.136372 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.136373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.137039 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.137042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.137439 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.137440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.137487 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.138119 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:57.138125 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:57.140292 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20260925 10:13:57.140506 12961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:57.135024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:51713 (local address 127.3.152.254:32851)
0925 10:13:57.135218 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:57.135220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:57.135231 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:57.135297 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:57.135298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:57.135309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:57.135365 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:57.135719 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.136275 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.137137 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.137358 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.137419 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.140413 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:51713: BlockingRecv error: recv got EOF from 127.3.152.193:51713 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:57.277534 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:57.272515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:37689)
0925 10:13:57.272679 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:57.272693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:57.272762 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:57.272910 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:57.272911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:57.273079 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:57.273135 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.273136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.273829 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.273829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.274365 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.274367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.274750 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.274751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.274798 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.275328 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:57.275337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:57.277473 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":168,"threads_started":1}
W20260925 10:13:57.277695 12961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:57.272704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:37689 (local address 127.3.152.254:32851)
0925 10:13:57.272756 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:57.272758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:57.272801 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:13:57.272830 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:57.272831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:57.272842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:57.272886 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:57.273219 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.273750 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.274448 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.274660 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.274719 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.277565 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:37689: BlockingRecv error: recv got EOF from 127.3.152.194:37689 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:57 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:57.412566 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:57.407428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:53595)
0925 10:13:57.407589 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:57.407602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:57.407692 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:57.407914 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:57.407915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:57.408098 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:57.408146 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.408147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.408856 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.408857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.409396 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:57.409398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.409748 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.409749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:57.409795 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.410273 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:57.410278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:57.412501 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
W20260925 10:13:57.412667 12961 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:57.407537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53595 (local address 127.3.152.254:32851)
0925 10:13:57.407666 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:57.407668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:57.407679 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:57.407768 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:57.407769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:57.407782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:57.407851 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:57.408223 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.408767 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.409466 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:57.409677 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:57.409730 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:57.412601 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:53595: BlockingRecv error: recv got EOF from 127.3.152.195:53595 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:58.150604 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:58.140931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:38207)
0925 10:13:58.141111 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:58.141122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:58.141219 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:58.141402 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:58.141402 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:58.141641 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:58.141706 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.141708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.142754 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.142755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.143379 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.143382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.143828 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.143829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.143889 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.148483 (+ 4594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:58.148490 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:58.150517 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260925 10:13:58.150776 12965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:58.140975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:38207 (local address 127.3.152.254:32851)
0925 10:13:58.141103 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:58.141106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:58.141218 (+ 112us) server_negotiation.cc:407] Connection header received
0925 10:13:58.141281 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:58.141283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:58.141299 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:58.141389 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:58.141796 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.142648 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.143450 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.143667 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.143812 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.150713 (+ 6901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:38207: BlockingRecv error: recv got EOF from 127.3.152.193:38207 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:58.283542 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:58.278112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:34963)
0925 10:13:58.278267 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:58.278279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:58.278390 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:58.278577 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:58.278578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:58.278821 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:58.278869 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.278872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.279622 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.279623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.280365 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.280367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.280760 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.280761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.280812 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.281419 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:58.281424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:58.283414 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260925 10:13:58.283708 12965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:58.278191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:34963 (local address 127.3.152.254:32851)
0925 10:13:58.278272 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:58.278274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:58.278368 (+ 94us) server_negotiation.cc:407] Connection header received
0925 10:13:58.278452 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:58.278453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:58.278466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:58.278524 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:58.278943 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.279512 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.280450 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.280677 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.280737 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.283581 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:34963: BlockingRecv error: recv got EOF from 127.3.152.194:34963 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:58 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.195@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:58.418498 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:58.413235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.195:54695)
0925 10:13:58.413399 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:58.413414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:58.413475 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:58.413602 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:58.413603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:58.413789 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:58.413837 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.413839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.414499 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.414500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.415060 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:58.415062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.415546 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.415547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:58.415603 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.416201 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:58.416206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:58.418376 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
W20260925 10:13:58.418581 12965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:58.413349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:54695 (local address 127.3.152.254:32851)
0925 10:13:58.413411 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:58.413413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:58.413463 (+ 50us) server_negotiation.cc:407] Connection header received
0925 10:13:58.413521 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:58.413522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:58.413534 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:58.413591 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:58.413908 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.414407 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.415215 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:58.415425 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:58.415498 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:58.418503 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.195:54695: BlockingRecv error: recv got EOF from 127.3.152.195:54695 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 10:13:59 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:59 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.193@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:59.157083 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.151329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.193:40291)
0925 10:13:59.151529 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:59.151541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:59.151606 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:59.151927 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:59.151928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:59.152144 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:59.152201 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.152203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.153046 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.153048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.153827 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.153830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.154392 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.154393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.154443 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.154939 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:59.154944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:59.157006 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":118,"threads_started":1}
W20260925 10:13:59.157238 12969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:59.151453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40291 (local address 127.3.152.254:32851)
0925 10:13:59.151600 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.151602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.151613 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:13:59.151790 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.151791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.151806 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.151883 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.152321 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.152931 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.153964 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.154306 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.154369 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.157123 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.193:40291: BlockingRecv error: recv got EOF from 127.3.152.193:40291 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
Sep 25 10:13:59 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 10:13:59 dist-test-slave-dzwq krb5kdc[11911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.152.194@KRBTEST.COM for oryx/127.3.152.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 10:13:59.289858 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.284290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:32851 (local address 127.3.152.194:45941)
0925 10:13:59.284633 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:59.284646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:59.284756 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:59.284969 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:59.284970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:59.285174 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:59.285227 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.285229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.285968 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.285969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.286534 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.286536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.287217 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.287218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.287294 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.287853 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:13:59.287858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:59.289796 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.152.254:32851: Server oryx/127.3.152.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
W20260925 10:13:59.290062 12969 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:59.284412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:45941 (local address 127.3.152.254:32851)
0925 10:13:59.284523 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.284526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.284734 (+ 208us) server_negotiation.cc:407] Connection header received
0925 10:13:59.284843 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.284845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.284874 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.284944 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.285326 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.285867 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.286703 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.287065 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.287190 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.289997 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.152.194:45941: BlockingRecv error: recv got EOF from 127.3.152.194:45941 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 10:13:59.331444 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12494
I20260925 10:13:59.336187 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12646
I20260925 10:13:59.340776 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12779
I20260925 10:13:59.345438 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12424
2026-09-25T10:13:59Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17540 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T10:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:59Z Disabled control of system clock
I20260925 10:13:59.373374 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46153
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:46153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 10:13:59.457387 12974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:59.457612 12974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:59.457634 12974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:59.458811 12974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:13:59.458863 12974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:59.458875 12974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:59.458887 12974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:13:59.458901 12974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:13:59.460400 12974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:46153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:59.460634 12974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:59.460913 12974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:59.466197 12979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:59.466301 12974 server_base.cc:1034] running on GCE node
W20260925 10:13:59.466207 12980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.466270 12982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:59.466706 12974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:59.467031 12974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:59.468210 12974 hybrid_clock.cc:648] HybridClock initialized: now 1790331239468188 us; error 41 us; skew 500 ppm
I20260925 10:13:59.469530 12974 webserver.cc:460] Webserver started at http://127.3.152.254:37059/ using document root <none> and password file <none>
I20260925 10:13:59.469842 12974 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:59.469894 12974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:59.470073 12974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:59.471220 12974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "1b119501243440a08f9e496843041727"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "d469a673e4a151005fe84fe7822814f0"
server_key_iv: "61b9ed1f8dc8adb56d6fa0655b42e6bb"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.471589 12974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "1b119501243440a08f9e496843041727"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "d469a673e4a151005fe84fe7822814f0"
server_key_iv: "61b9ed1f8dc8adb56d6fa0655b42e6bb"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.473052 12974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:13:59.473882 12988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.474066 12974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:59.474182 12974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "1b119501243440a08f9e496843041727"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "d469a673e4a151005fe84fe7822814f0"
server_key_iv: "61b9ed1f8dc8adb56d6fa0655b42e6bb"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.474259 12974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:59.484460 12974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:59.484822 12974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:59.484992 12974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:59.489109 12974 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:46153
I20260925 10:13:59.489146 13040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:46153 every 8 connection(s)
I20260925 10:13:59.489531 12974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:13:59.490319 13041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:13:59.493119 13041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727: Bootstrap starting.
I20260925 10:13:59.493750 13041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727: Neither blocks nor log segments found. Creating new log.
I20260925 10:13:59.494061 13041 log.cc:824] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727: Log is configured to *not* fsync() on all Append() calls
I20260925 10:13:59.494833 13041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727: No bootstrap required, opened a new log
I20260925 10:13:59.496507 13041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } }
I20260925 10:13:59.496647 13041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:13:59.496670 13041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b119501243440a08f9e496843041727, State: Initialized, Role: FOLLOWER
I20260925 10:13:59.496856 13041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } }
I20260925 10:13:59.496920 13041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:13:59.496963 13041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:13:59.497010 13041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:13:59.497547 13041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } }
I20260925 10:13:59.497659 13041 leader_election.cc:302] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b119501243440a08f9e496843041727; no voters:
I20260925 10:13:59.497855 13041 leader_election.cc:288] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:13:59.497921 13044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:13:59.497982 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 12974
I20260925 10:13:59.498087 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:13:59.498134 13041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:13:59.498178 13044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [term 1 LEADER]: Becoming Leader. State: Replica: 1b119501243440a08f9e496843041727, State: Running, Role: LEADER
I20260925 10:13:59.498292 3683 external_mini_cluster.cc:1428] Setting key fe438c59ce8b7b2a75c265cda8023eda
I20260925 10:13:59.498315 13044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } }
I20260925 10:13:59.498968 13045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b119501243440a08f9e496843041727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } } }
I20260925 10:13:59.499060 13045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:59.499297 13055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:13:59.499524 13046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b119501243440a08f9e496843041727. Latest consensus state: current_term: 1 leader_uuid: "1b119501243440a08f9e496843041727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b119501243440a08f9e496843041727" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46153 } } }
I20260925 10:13:59.499673 13046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727 [sys.catalog]: This master's current role is: LEADER
I20260925 10:13:59.499910 13055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:13:59.501747 13055 catalog_manager.cc:1347] Generated new cluster ID: 149ad90e6d5c44f6b1f0855cea983e9a
I20260925 10:13:59.501803 13055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:13:59.505223 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.499001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.3.152.254:46153)
0925 10:13:59.499820 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.499829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.499859 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:13:59.500005 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.500015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.500090 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.500385 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.500564 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.501826 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.502983 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.503595 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.503715 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.504452 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:59.504456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:59.504457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:59.504685 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:59.504726 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:59.504740 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:59.504800 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:59.504914 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 10:13:59.504998 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":78,"threads_started":1}
I20260925 10:13:59.514539 13055 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:13:59.515327 13055 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:13:59.521430 13055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b119501243440a08f9e496843041727: Generated new TSK 0
I20260925 10:13:59.521723 13055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:13:59.524709 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--builtin_ntp_servers=127.3.152.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 10:13:59.613467 13065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:59.613705 13065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:59.613730 13065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:59.614993 13065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:59.615041 13065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:59.615068 13065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:13:59.616514 13065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:59.616811 13065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:59.617130 13065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:59.622416 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.622421 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.622417 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:59.622783 13065 server_base.cc:1034] running on GCE node
I20260925 10:13:59.623036 13065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:59.623330 13065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:59.624590 13065 hybrid_clock.cc:648] HybridClock initialized: now 1790331239624507 us; error 42 us; skew 500 ppm
I20260925 10:13:59.626081 13065 webserver.cc:460] Webserver started at http://127.3.152.193:37561/ using document root <none> and password file <none>
I20260925 10:13:59.626340 13065 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:59.626392 13065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:59.626569 13065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:59.627862 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "8155c40afe6d4edbbc95ff35440848ec"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "cbeb7bc77bd9d75908eb5f687add1bbe"
server_key_iv: "fd6d5f5d0b7a12914c6f84a9613027af"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.628340 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "8155c40afe6d4edbbc95ff35440848ec"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "cbeb7bc77bd9d75908eb5f687add1bbe"
server_key_iv: "fd6d5f5d0b7a12914c6f84a9613027af"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.629709 13065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:59.630501 13079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.630746 13065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:59.630860 13065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "8155c40afe6d4edbbc95ff35440848ec"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "cbeb7bc77bd9d75908eb5f687add1bbe"
server_key_iv: "fd6d5f5d0b7a12914c6f84a9613027af"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.630954 13065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:59.655524 13065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:59.655997 13065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:59.656167 13065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:59.656451 13065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:59.656836 13065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:59.656898 13065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.656942 13065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:59.656991 13065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.663841 13065 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:43191
I20260925 10:13:59.663887 13192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:43191 every 8 connection(s)
I20260925 10:13:59.664311 13065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:13:59.670050 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13065
I20260925 10:13:59.670163 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:13:59.670233 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.665206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:38993 (local address 127.3.152.254:46153)
0925 10:13:59.665274 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.665277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.665695 (+ 418us) server_negotiation.cc:407] Connection header received
0925 10:13:59.666272 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.666273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.666289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.666351 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.667043 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.667545 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.668732 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.668932 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.668997 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.669868 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:59.669869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:59.669869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:59.669874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:59.669893 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:59.669897 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:59.669960 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:59.670076 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:13:59.670107 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:13:59.670378 3683 external_mini_cluster.cc:1428] Setting key e1c151ed51f3fd7322c1754250f73194
I20260925 10:13:59.670446 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.665216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:46153 (local address 127.3.152.193:38993)
0925 10:13:59.665486 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:59.665525 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:59.666147 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:59.666409 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:59.666413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:59.666421 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:59.666949 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.666954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.667658 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.667659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.668560 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.668562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.669018 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.669019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.669138 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.669704 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:59.669705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:59.669735 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:59.669760 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:59.669978 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:59.669985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:59.669989 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:59.670071 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:13:59.670196 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260925 10:13:59.671156 13193 heartbeater.cc:347] Connected to a master server at 127.3.152.254:46153
I20260925 10:13:59.671245 13193 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:59.671437 13193 heartbeater.cc:511] Master 127.3.152.254:46153 requested a full tablet report, sending...
I20260925 10:13:59.671446 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--builtin_ntp_servers=127.3.152.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:13:59.671955 13005 ts_manager.cc:194] Registered new tserver with Master: 8155c40afe6d4edbbc95ff35440848ec (127.3.152.193:43191)
I20260925 10:13:59.672626 13005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.193:38993
W20260925 10:13:59.754088 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:59.754321 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:59.754343 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:59.755471 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:59.755513 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:59.755538 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:13:59.756959 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:59.757162 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:59.757444 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:59.762580 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.762656 13204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.762564 13202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:59.762698 13196 server_base.cc:1034] running on GCE node
I20260925 10:13:59.763074 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:59.763283 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:59.764505 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790331239764430 us; error 88 us; skew 500 ppm
I20260925 10:13:59.766243 13196 webserver.cc:460] Webserver started at http://127.3.152.194:42297/ using document root <none> and password file <none>
I20260925 10:13:59.766490 13196 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:59.766559 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:59.766739 13196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:59.767856 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "527cc9bda84141218671cac0eb90a771"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "b8e2b26300a4963d475a36e1edcefbbf"
server_key_iv: "eadd75b69a319fd45025d4476bf68e18"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.768316 13196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "527cc9bda84141218671cac0eb90a771"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "b8e2b26300a4963d475a36e1edcefbbf"
server_key_iv: "eadd75b69a319fd45025d4476bf68e18"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.769724 13196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:59.770484 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.770646 13196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:13:59.770746 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "527cc9bda84141218671cac0eb90a771"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "b8e2b26300a4963d475a36e1edcefbbf"
server_key_iv: "eadd75b69a319fd45025d4476bf68e18"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.770830 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:59.785068 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:59.785431 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:59.785606 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:59.785866 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:59.786224 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:59.786283 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.786329 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:59.786352 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.792356 13196 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:33517
I20260925 10:13:59.792412 13323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:33517 every 8 connection(s)
I20260925 10:13:59.792807 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:13:59.796754 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13196
I20260925 10:13:59.796849 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:13:59.797060 3683 external_mini_cluster.cc:1428] Setting key 92c898492a8ebc176d701ccbc7e4d195
I20260925 10:13:59.798326 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--builtin_ntp_servers=127.3.152.212:32845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 10:13:59.798951 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.793601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:51981 (local address 127.3.152.254:46153)
0925 10:13:59.793655 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.793657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.793994 (+ 337us) server_negotiation.cc:407] Connection header received
0925 10:13:59.794543 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.794544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.794560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.794622 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.795327 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.795838 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.797149 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.797452 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.797531 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.798639 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:59.798640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:59.798640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:59.798651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:59.798667 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:59.798671 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:59.798752 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:59.798836 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 10:13:59.798872 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:13:59.799181 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.793577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:46153 (local address 127.3.152.194:51981)
0925 10:13:59.793824 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:59.793896 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:59.794432 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:59.794706 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:59.794711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:59.794721 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:59.795231 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.795236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.795938 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.795940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.797021 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.797025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.797555 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.797556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.797711 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.798423 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:59.798424 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:59.798468 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:59.798541 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:59.798752 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:59.798759 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:59.798762 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:13:59.798839 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 10:13:59.798950 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260925 10:13:59.800117 13324 heartbeater.cc:347] Connected to a master server at 127.3.152.254:46153
I20260925 10:13:59.800257 13324 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:59.800452 13324 heartbeater.cc:511] Master 127.3.152.254:46153 requested a full tablet report, sending...
I20260925 10:13:59.800899 13005 ts_manager.cc:194] Registered new tserver with Master: 527cc9bda84141218671cac0eb90a771 (127.3.152.194:33517)
I20260925 10:13:59.801304 13005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.194:51981
W20260925 10:13:59.883908 13327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:13:59.884171 13327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:13:59.884205 13327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:13:59.885419 13327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:13:59.885486 13327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:13:59.885526 13327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:13:59.886899 13327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:32845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:46153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:13:59.887176 13327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:13:59.887490 13327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:13:59.893121 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.893121 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:13:59.893213 13335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:13:59.893362 13327 server_base.cc:1034] running on GCE node
I20260925 10:13:59.893700 13327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:13:59.894001 13327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:13:59.895200 13327 hybrid_clock.cc:648] HybridClock initialized: now 1790331239895176 us; error 47 us; skew 500 ppm
I20260925 10:13:59.896699 13327 webserver.cc:460] Webserver started at http://127.3.152.195:34829/ using document root <none> and password file <none>
I20260925 10:13:59.896952 13327 fs_manager.cc:359] Metadata directory not provided
I20260925 10:13:59.897023 13327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:13:59.897179 13327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:13:59.898362 13327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "9ad5882dec3c4f6fba78ec16c580f214"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "2486718f0cd47ca58fae1d2e2e07fdaa"
server_key_iv: "0249025905ce531db53b57af0aeda4f9"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.898774 13327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "9ad5882dec3c4f6fba78ec16c580f214"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "2486718f0cd47ca58fae1d2e2e07fdaa"
server_key_iv: "0249025905ce531db53b57af0aeda4f9"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.900254 13327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 10:13:59.901068 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.901274 13327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:13:59.901381 13327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "9ad5882dec3c4f6fba78ec16c580f214"
format_stamp: "Formatted at 2026-09-25 10:13:59 on dist-test-slave-dzwq"
server_key: "2486718f0cd47ca58fae1d2e2e07fdaa"
server_key_iv: "0249025905ce531db53b57af0aeda4f9"
server_key_version: "encryptionkey@0"
I20260925 10:13:59.901455 13327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:13:59.931093 13327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:13:59.931463 13327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:13:59.931656 13327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:13:59.931915 13327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:13:59.932312 13327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:13:59.932370 13327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.932415 13327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:13:59.932461 13327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:13:59.938529 13327 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:46543
I20260925 10:13:59.938575 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:46543 every 8 connection(s)
I20260925 10:13:59.938930 13327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:13:59.944339 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13327
I20260925 10:13:59.944453 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:13:59.944723 3683 external_mini_cluster.cc:1428] Setting key 0eac5ba526fe568fa5843704042dd780
I20260925 10:13:59.944806 13054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.939694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:48355 (local address 127.3.152.254:46153)
0925 10:13:59.939767 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.939770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.940014 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:13:59.940534 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.940535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.940544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.940658 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.941443 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.942005 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.943056 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.943243 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.943305 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.944396 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:13:59.944398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:13:59.944398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:13:59.944403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:13:59.944418 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:13:59.944422 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:13:59.944496 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:13:59.944681 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 10:13:59.944714 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:13:59.944947 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:13:59.939680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:46153 (local address 127.3.152.195:48355)
0925 10:13:59.939868 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:13:59.939913 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:13:59.940444 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:13:59.940818 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:13:59.940822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:13:59.940831 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:13:59.941335 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.941339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.942100 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.942101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.942965 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:13:59.942967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:13:59.943345 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:13:59.943347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:13:59.943468 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:13:59.944237 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:13:59.944238 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:13:59.944268 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:13:59.944291 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:13:59.944521 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:13:59.944527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:13:59.944531 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:13:59.944612 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 10:13:59.944700 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 10:13:59.945391 13455 heartbeater.cc:347] Connected to a master server at 127.3.152.254:46153
I20260925 10:13:59.945495 13455 heartbeater.cc:464] Registering TS with master...
I20260925 10:13:59.945712 13455 heartbeater.cc:511] Master 127.3.152.254:46153 requested a full tablet report, sending...
I20260925 10:13:59.946189 13005 ts_manager.cc:194] Registered new tserver with Master: 9ad5882dec3c4f6fba78ec16c580f214 (127.3.152.195:46543)
I20260925 10:13:59.946589 13005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.195:48355
I20260925 10:13:59.957031 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 10:13:59.959430 13054 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:13:59.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.3.152.254:46153)
0925 10:13:59.958936 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 10:13:59.958939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:13:59.959009 (+ 70us) server_negotiation.cc:407] Connection header received
0925 10:13:59.959071 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:13:59.959071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:13:59.959081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:13:59.959137 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:13:59.959374 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46838: BlockingRecv error: recv got EOF from 127.0.0.1:46838 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:13:59.959491 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13065
I20260925 10:13:59.964336 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13196
I20260925 10:13:59.968607 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13327
I20260925 10:13:59.972949 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 12974
2026-09-25T10:13:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (626 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T10:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:13:59Z Disabled control of system clock
I20260925 10:13:59.999764 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33935
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:14:00.082849 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:00.083076 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:00.083099 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:00.084370 13466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:14:00.084414 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:00.084427 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:00.084440 13466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:14:00.084452 13466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:14:00.085807 13466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:00.086017 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:00.086256 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:00.091485 13474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.091490 13472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.091522 13466 server_base.cc:1034] running on GCE node
W20260925 10:14:00.091487 13471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.092077 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:00.092394 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:00.093569 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790331240093556 us; error 37 us; skew 500 ppm
I20260925 10:14:00.095096 13466 webserver.cc:460] Webserver started at http://127.3.152.254:39045/ using document root <none> and password file <none>
I20260925 10:14:00.095355 13466 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:00.095428 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:00.095583 13466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:00.096743 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "4d544d5bd9a44563951cae7ba6b7bd35"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "b4d5b4edbdc02701513990bf458b96e6"
server_key_iv: "0da28d0651f479e86225f2c901221512"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.097110 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "4d544d5bd9a44563951cae7ba6b7bd35"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "b4d5b4edbdc02701513990bf458b96e6"
server_key_iv: "0da28d0651f479e86225f2c901221512"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.098485 13466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:14:00.099368 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.099545 13466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:00.099727 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "4d544d5bd9a44563951cae7ba6b7bd35"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "b4d5b4edbdc02701513990bf458b96e6"
server_key_iv: "0da28d0651f479e86225f2c901221512"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.099812 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:00.103292 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:00.103612 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:00.103766 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:00.107873 13466 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:33935
I20260925 10:14:00.107906 13532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:33935 every 8 connection(s)
I20260925 10:14:00.108280 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:14:00.109001 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:00.112103 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35: Bootstrap starting.
I20260925 10:14:00.112762 13533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:00.113101 13533 log.cc:824] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:00.113864 13533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35: No bootstrap required, opened a new log
I20260925 10:14:00.114708 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13466
I20260925 10:14:00.114789 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:14:00.114928 3683 external_mini_cluster.cc:1428] Setting key 9eff9ec797ea0d2b7b13ba956fa1bccc
I20260925 10:14:00.116429 13533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } }
I20260925 10:14:00.116605 13533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:00.116643 13533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d544d5bd9a44563951cae7ba6b7bd35, State: Initialized, Role: FOLLOWER
I20260925 10:14:00.116801 13533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } }
I20260925 10:14:00.116925 13533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:14:00.116967 13533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:14:00.117009 13533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:00.117836 13533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } }
I20260925 10:14:00.117961 13533 leader_election.cc:302] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d544d5bd9a44563951cae7ba6b7bd35; no voters:
I20260925 10:14:00.118018 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.115640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.3.152.254:33935)
0925 10:14:00.115857 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:00.115865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:00.115894 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:14:00.116415 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:00.116423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:00.116469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:00.116661 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:00.117423 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:00.117426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:00.117426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:14:00.117430 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:00.117469 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:14:00.117481 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:00.117549 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:00.117731 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:14:00.117806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260925 10:14:00.118137 13533 leader_election.cc:288] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:14:00.118403 13533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:14:00.118650 13539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:14:00.118808 13539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [term 1 LEADER]: Becoming Leader. State: Replica: 4d544d5bd9a44563951cae7ba6b7bd35, State: Running, Role: LEADER
I20260925 10:14:00.118937 13539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } }
I20260925 10:14:00.119335 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d544d5bd9a44563951cae7ba6b7bd35. Latest consensus state: current_term: 1 leader_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } } }
I20260925 10:14:00.119374 13538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d544d5bd9a44563951cae7ba6b7bd35" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33935 } } }
I20260925 10:14:00.119550 13538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:00.119515 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:00.119908 13553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:14:00.120491 13553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:14:00.122200 13553 catalog_manager.cc:1347] Generated new cluster ID: 436f748cf330462680e19eb25765e930
I20260925 10:14:00.122262 13553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:14:00.141265 13553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:14:00.141494 13553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:14:00.152511 13553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d544d5bd9a44563951cae7ba6b7bd35: Generated new TSK 0
I20260925 10:14:00.152776 13553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:14:00.156311 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--builtin_ntp_servers=127.3.152.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:14:00.241485 13557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:00.241775 13557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:00.241817 13557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:00.243101 13557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:00.243189 13557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:00.243237 13557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:14:00.244830 13557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:00.245141 13557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:00.245481 13557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:00.251569 13562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.251610 13563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.251610 13565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.251881 13557 server_base.cc:1034] running on GCE node
I20260925 10:14:00.252175 13557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:00.252417 13557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:00.253590 13557 hybrid_clock.cc:648] HybridClock initialized: now 1790331240253573 us; error 30 us; skew 500 ppm
I20260925 10:14:00.255213 13557 webserver.cc:460] Webserver started at http://127.3.152.193:37199/ using document root <none> and password file <none>
I20260925 10:14:00.255489 13557 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:00.255578 13557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:00.255771 13557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:00.257082 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "2af9190cdfbb4bcb9af05bb373362c6e"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "2e9524513f02736247a54ec73fb4837d"
server_key_iv: "a17c5b526a1aec9956f50aa3e10bf8f7"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.257525 13557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "2af9190cdfbb4bcb9af05bb373362c6e"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "2e9524513f02736247a54ec73fb4837d"
server_key_iv: "a17c5b526a1aec9956f50aa3e10bf8f7"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.259068 13557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:14:00.260067 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.260326 13557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:00.260438 13557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "2af9190cdfbb4bcb9af05bb373362c6e"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "2e9524513f02736247a54ec73fb4837d"
server_key_iv: "a17c5b526a1aec9956f50aa3e10bf8f7"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.260537 13557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:00.264364 13557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:00.264662 13557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:00.264825 13557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:00.265053 13557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:00.265410 13557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:00.265470 13557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.265512 13557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:00.265542 13557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.271611 13557 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:46269
I20260925 10:14:00.271721 13684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:46269 every 8 connection(s)
I20260925 10:14:00.272179 13557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:14:00.275441 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.273022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:35167 (local address 127.3.152.254:33935)
0925 10:14:00.273094 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:00.273097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:00.273358 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:14:00.273889 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:00.273890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:00.273897 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:00.273952 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:00.275223 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:00.275224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:00.275225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:14:00.275228 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:00.275245 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:14:00.275249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:00.275307 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:00.275368 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 10:14:00.275382 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:14:00.275646 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.273019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33935 (local address 127.3.152.193:35167)
0925 10:14:00.273219 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:00.273255 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:00.273794 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:00.274019 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:00.274022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:00.274031 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:00.275043 (+ 1012us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:14:00.275043 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:00.275075 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:14:00.275107 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:00.275305 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:00.275308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:00.275310 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:14:00.275366 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:14:00.275418 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260925 10:14:00.276234 13685 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33935
I20260925 10:14:00.276333 13685 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:00.276454 13685 heartbeater.cc:511] Master 127.3.152.254:33935 requested a full tablet report, sending...
I20260925 10:14:00.277014 13497 ts_manager.cc:194] Registered new tserver with Master: 2af9190cdfbb4bcb9af05bb373362c6e (127.3.152.193:46269)
I20260925 10:14:00.281929 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13557
I20260925 10:14:00.282014 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:14:00.282249 3683 external_mini_cluster.cc:1428] Setting key 04bf0e7b152859486d8f64ed159ea957
I20260925 10:14:00.283439 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--builtin_ntp_servers=127.3.152.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 10:14:00.367316 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:00.367523 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:00.367545 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:00.368966 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:00.369010 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:00.369035 13688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:14:00.370383 13688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:00.370649 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:00.370956 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:00.376415 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.376421 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.376457 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.376570 13688 server_base.cc:1034] running on GCE node
I20260925 10:14:00.376853 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:00.377077 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:00.378257 13688 hybrid_clock.cc:648] HybridClock initialized: now 1790331240378206 us; error 38 us; skew 500 ppm
I20260925 10:14:00.379698 13688 webserver.cc:460] Webserver started at http://127.3.152.194:39377/ using document root <none> and password file <none>
I20260925 10:14:00.380025 13688 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:00.380093 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:00.380294 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:00.381505 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "7fec52a9dbd94b50904c0bc94b40ede9"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "c3dbeb9ea68bbca083fb186503870e0a"
server_key_iv: "19dfbc5fa501851a4163dc00b1e312ba"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.381924 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "7fec52a9dbd94b50904c0bc94b40ede9"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "c3dbeb9ea68bbca083fb186503870e0a"
server_key_iv: "19dfbc5fa501851a4163dc00b1e312ba"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.383397 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:14:00.384387 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.384599 13688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:14:00.384739 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "7fec52a9dbd94b50904c0bc94b40ede9"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "c3dbeb9ea68bbca083fb186503870e0a"
server_key_iv: "19dfbc5fa501851a4163dc00b1e312ba"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.384824 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:00.388588 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:00.388897 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:00.389055 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:00.389282 13688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:00.389626 13688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:00.389660 13688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.389714 13688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:00.389743 13688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.396214 13688 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:43397
I20260925 10:14:00.396309 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:43397 every 8 connection(s)
I20260925 10:14:00.396672 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:14:00.398777 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13688
I20260925 10:14:00.398918 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:14:00.399140 3683 external_mini_cluster.cc:1428] Setting key e9f1c1b48ca1968aa9d1324f29ad2420
I20260925 10:14:00.400306 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.397651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:59805 (local address 127.3.152.254:33935)
0925 10:14:00.397773 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:00.397775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:00.397972 (+ 197us) server_negotiation.cc:407] Connection header received
0925 10:14:00.398520 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:00.398522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:00.398529 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:00.398596 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:00.399998 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:00.400000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:00.400000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:14:00.400006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:00.400025 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:14:00.400030 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:00.400094 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:00.400212 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:14:00.400229 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 10:14:00.400324 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--builtin_ntp_servers=127.3.152.212:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 10:14:00.400539 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.397559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33935 (local address 127.3.152.194:59805)
0925 10:14:00.397842 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:00.397880 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:00.398406 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:00.398692 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:00.398695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:00.398705 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:00.399800 (+ 1095us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:14:00.399800 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:00.399833 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:14:00.399866 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:00.400108 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:00.400114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:00.400118 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:14:00.400224 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:14:00.400295 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":131,"threads_started":1}
I20260925 10:14:00.401060 13816 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33935
I20260925 10:14:00.401185 13816 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:00.401364 13816 heartbeater.cc:511] Master 127.3.152.254:33935 requested a full tablet report, sending...
I20260925 10:14:00.401885 13497 ts_manager.cc:194] Registered new tserver with Master: 7fec52a9dbd94b50904c0bc94b40ede9 (127.3.152.194:43397)
W20260925 10:14:00.483932 13819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:00.484309 13819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:00.484352 13819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:00.485632 13819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:00.485709 13819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:00.485755 13819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:14:00.487154 13819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:00.487455 13819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:00.487855 13819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:00.493053 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:00.493093 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.493314 13819 server_base.cc:1034] running on GCE node
W20260925 10:14:00.493121 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:00.493635 13819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:00.493903 13819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:00.495085 13819 hybrid_clock.cc:648] HybridClock initialized: now 1790331240495086 us; error 44 us; skew 500 ppm
I20260925 10:14:00.496520 13819 webserver.cc:460] Webserver started at http://127.3.152.195:41429/ using document root <none> and password file <none>
I20260925 10:14:00.496790 13819 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:00.496865 13819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:00.497031 13819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:00.498180 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "88063961c3374444a513711133c6d253"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "d8eceac23f3521e4edb8c8779180459d"
server_key_iv: "9f878aa94c5fa079afcc1e7d9bcc70df"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.498718 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "88063961c3374444a513711133c6d253"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "d8eceac23f3521e4edb8c8779180459d"
server_key_iv: "9f878aa94c5fa079afcc1e7d9bcc70df"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.500243 13819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:14:00.501039 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.501245 13819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:00.501353 13819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "88063961c3374444a513711133c6d253"
format_stamp: "Formatted at 2026-09-25 10:14:00 on dist-test-slave-dzwq"
server_key: "d8eceac23f3521e4edb8c8779180459d"
server_key_iv: "9f878aa94c5fa079afcc1e7d9bcc70df"
server_key_version: "encryptionkey@0"
I20260925 10:14:00.501431 13819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:00.505156 13819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:00.505435 13819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:00.505576 13819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:00.505782 13819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:00.506131 13819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:00.506186 13819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.506225 13819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:00.506253 13819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:00.512069 13819 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:42577
I20260925 10:14:00.512157 13946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:42577 every 8 connection(s)
I20260925 10:14:00.512563 13819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:14:00.515681 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.513363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:33807 (local address 127.3.152.254:33935)
0925 10:14:00.513431 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:00.513433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:00.513799 (+ 366us) server_negotiation.cc:407] Connection header received
0925 10:14:00.514257 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:00.514258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:00.514263 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:00.514317 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:00.515437 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:00.515438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:00.515438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:14:00.515442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:00.515455 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:14:00.515459 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:00.515508 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:00.515568 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 10:14:00.515581 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:14:00.515902 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:00.513367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33935 (local address 127.3.152.195:33807)
0925 10:14:00.513578 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:00.513705 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:00.514170 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:00.514373 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:00.514376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:00.514383 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:00.515274 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:14:00.515274 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:00.515303 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:14:00.515338 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:00.515505 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:00.515508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:00.515510 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:14:00.515562 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:14:00.515658 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260925 10:14:00.516222 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13819
I20260925 10:14:00.516317 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:14:00.516515 3683 external_mini_cluster.cc:1428] Setting key f2c6c0e8151f0bcec792e25dbbaa6fb7
I20260925 10:14:00.517086 13947 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33935
I20260925 10:14:00.517177 13947 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:00.517342 13947 heartbeater.cc:511] Master 127.3.152.254:33935 requested a full tablet report, sending...
I20260925 10:14:00.517753 13496 ts_manager.cc:194] Registered new tserver with Master: 88063961c3374444a513711133c6d253 (127.3.152.195:42577)
I20260925 10:14:00.528508 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:14:00.530601 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13557
W20260925 10:14:00.530658 13537 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:14:00.530000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.3.152.254:33935)
0925 10:14:00.530095 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:00.530098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:00.530219 (+ 121us) server_negotiation.cc:407] Connection header received
0925 10:14:00.530301 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:00.530302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:00.530307 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:00.530358 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:00.530495 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59758: BlockingRecv error: recv got EOF from 127.0.0.1:59758 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 10:14:00.534982 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13688
I20260925 10:14:00.539664 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13819
I20260925 10:14:00.544219 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13466
2026-09-25T10:14:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (571 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:14:00 dist-test-slave-dzwq krb5kdc[13957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:14:00 dist-test-slave-dzwq krb5kdc[13957](info): set up 2 sockets
Sep 25 10:14:00 dist-test-slave-dzwq krb5kdc[13957](info): commencing operation
krb5kdc: starting...
W20260925 10:14:01.589654 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:14:01 dist-test-slave-dzwq krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:14:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:14:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:14:01.718847 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:45019
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:01.802058 13970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:01.802352 13970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:01.802390 13970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:01.803598 13970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:14:01.803683 13970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:01.803707 13970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:01.803728 13970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:14:01.803759 13970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:14:01.805339 13970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:01.805655 13970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:01.805900 13970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:01.811266 13976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:01.811293 13978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:01.811326 13970 server_base.cc:1034] running on GCE node
W20260925 10:14:01.811290 13975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:01.811902 13970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:01.812250 13970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:01.813450 13970 hybrid_clock.cc:648] HybridClock initialized: now 1790331241813428 us; error 47 us; skew 500 ppm
Sep 25 10:14:01 dist-test-slave-dzwq krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331241, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:01.816659 13970 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:14:01.817198 13970 webserver.cc:460] Webserver started at http://127.3.152.254:38675/ using document root <none> and password file <none>
I20260925 10:14:01.817489 13970 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:01.817560 13970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:01.817714 13970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:01.819195 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "7a55649ce90b43f2b755697531d8ee65"
format_stamp: "Formatted at 2026-09-25 10:14:01 on dist-test-slave-dzwq"
server_key: "6bc3d33a4086c97c0ca7ae628d8ae20d"
server_key_iv: "8a1678102f8892fc07b980147dff41e1"
server_key_version: "encryptionkey@0"
I20260925 10:14:01.819634 13970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "7a55649ce90b43f2b755697531d8ee65"
format_stamp: "Formatted at 2026-09-25 10:14:01 on dist-test-slave-dzwq"
server_key: "6bc3d33a4086c97c0ca7ae628d8ae20d"
server_key_iv: "8a1678102f8892fc07b980147dff41e1"
server_key_version: "encryptionkey@0"
I20260925 10:14:01.821357 13970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:14:01.822249 13985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:01.822430 13970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:01.822537 13970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "7a55649ce90b43f2b755697531d8ee65"
format_stamp: "Formatted at 2026-09-25 10:14:01 on dist-test-slave-dzwq"
server_key: "6bc3d33a4086c97c0ca7ae628d8ae20d"
server_key_iv: "8a1678102f8892fc07b980147dff41e1"
server_key_version: "encryptionkey@0"
I20260925 10:14:01.822695 13970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:01.855773 13970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:01.858036 13970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:01.858219 13970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:01.862466 13970 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:45019
I20260925 10:14:01.862505 14037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:45019 every 8 connection(s)
I20260925 10:14:01.862886 13970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:14:01.864046 14038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:01.864503 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 13970
I20260925 10:14:01.864622 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:14:01.864816 3683 external_mini_cluster.cc:1428] Setting key 41e9f9106aace356268d8448a7a0c827
I20260925 10:14:01.867111 14038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: Bootstrap starting.
I20260925 10:14:01.867750 14038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:01.868152 14038 log.cc:824] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:01.868903 14038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: No bootstrap required, opened a new log
I20260925 10:14:01.870841 14038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } }
I20260925 10:14:01.871049 14038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:01.871112 14038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a55649ce90b43f2b755697531d8ee65, State: Initialized, Role: FOLLOWER
I20260925 10:14:01.871258 14038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } }
I20260925 10:14:01.871368 14038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:14:01.871414 14038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:14:01.871471 14038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 10:14:01 dist-test-slave-dzwq krb5kdc[13957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:01.872327 14038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } }
I20260925 10:14:01.872440 14038 leader_election.cc:302] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a55649ce90b43f2b755697531d8ee65; no voters:
I20260925 10:14:01.872673 14038 leader_election.cc:288] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:14:01.872706 14043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:14:01.872838 14043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [term 1 LEADER]: Becoming Leader. State: Replica: 7a55649ce90b43f2b755697531d8ee65, State: Running, Role: LEADER
I20260925 10:14:01.872918 14043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } }
I20260925 10:14:01.872972 14038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:14:01.873238 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a55649ce90b43f2b755697531d8ee65. Latest consensus state: current_term: 1 leader_uuid: "7a55649ce90b43f2b755697531d8ee65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } } }
I20260925 10:14:01.873342 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:01.873423 14044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a55649ce90b43f2b755697531d8ee65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a55649ce90b43f2b755697531d8ee65" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 45019 } } }
I20260925 10:14:01.873524 14044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:01.873555 14048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:14:01.874121 14048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:14:01.876259 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:01.865388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.3.152.254:45019)
0925 10:14:01.865615 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:01.865623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:01.865651 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:14:01.866091 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:01.866099 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:01.866154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:01.866316 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:01.867030 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:01.868101 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:01.868897 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:01.869394 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:01.869519 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:01.872370 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:01.872374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:01.872375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:01.872386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:01.874650 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:01.875072 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:01.875075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:01.875075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:01.875157 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:01.875400 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:01.875401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:01.875401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:01.875617 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:01.875722 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:01.875878 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:14:01.876060 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260925 10:14:01.877171 14048 catalog_manager.cc:1347] Generated new cluster ID: ea1fe16ec5934589801176d75b6288cc
I20260925 10:14:01.877219 14048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:14:01.894842 14048 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:14:01.895929 14048 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:14:01.903479 14048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: Generated new TSK 0
I20260925 10:14:01.903724 14048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:14:01.942740 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:02.044487 14064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:02.044716 14064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:02.044744 14064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:02.045868 14064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:02.045907 14064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:02.045933 14064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:14:02.047346 14064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:02.047545 14064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:02.047731 14064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:02.053113 14070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:02.053146 14064 server_base.cc:1034] running on GCE node
W20260925 10:14:02.053113 14069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:02.053112 14072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:02.053524 14064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:02.053750 14064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:02.054912 14064 hybrid_clock.cc:648] HybridClock initialized: now 1790331242054902 us; error 27 us; skew 500 ppm
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:02.057533 14064 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:14:02.057996 14064 webserver.cc:460] Webserver started at http://127.3.152.193:45947/ using document root <none> and password file <none>
I20260925 10:14:02.058248 14064 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:02.058295 14064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:02.058471 14064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:02.059587 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "1d3ff9bd036c40fe8c03ef23da4bd058"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "ef15e6df2a5f02389280575adc24f5b4"
server_key_iv: "8237798556808f65b8caf9aa3fda02e3"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.059957 14064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "1d3ff9bd036c40fe8c03ef23da4bd058"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "ef15e6df2a5f02389280575adc24f5b4"
server_key_iv: "8237798556808f65b8caf9aa3fda02e3"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.061417 14064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:02.062321 14079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.062585 14064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:02.062680 14064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "1d3ff9bd036c40fe8c03ef23da4bd058"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "ef15e6df2a5f02389280575adc24f5b4"
server_key_iv: "8237798556808f65b8caf9aa3fda02e3"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.062757 14064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:02.076642 14064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:02.079018 14064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:02.079210 14064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:02.079463 14064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:02.079803 14064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:02.079861 14064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.079905 14064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:02.079936 14064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.085824 14064 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:41617
I20260925 10:14:02.085904 14192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:41617 every 8 connection(s)
I20260925 10:14:02.086254 14064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:14:02.088939 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 14064
I20260925 10:14:02.089037 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:14:02.089270 3683 external_mini_cluster.cc:1428] Setting key c53fccf500752812b8aa7d70f60edf9e
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:02.097600 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.087169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:54481 (local address 127.3.152.254:45019)
0925 10:14:02.087261 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.087263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.087445 (+ 182us) server_negotiation.cc:407] Connection header received
0925 10:14:02.088002 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.088004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.088017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.088089 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:02.089166 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.089804 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.091061 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.091392 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.091462 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.093938 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:02.093940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:02.093940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:02.093951 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:02.095870 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.096389 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.096391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.096391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.096460 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.096796 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.096797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.096798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.096957 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:02.097032 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.097348 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.097464 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:14:02.097616 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.087120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:45019 (local address 127.3.152.193:54481)
0925 10:14:02.087316 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.087355 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.087859 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.088194 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.088198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.088803 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:02.089026 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.089032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.089925 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.089926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.090871 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.090874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.091498 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.091500 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.091744 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.092331 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:02.092341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:02.093833 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:02.096015 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.096030 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.096032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.096303 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.096560 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.096561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.096562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.096717 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.097028 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:02.097034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:02.097189 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.097292 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.097431 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:14:02.098232 14193 heartbeater.cc:347] Connected to a master server at 127.3.152.254:45019
I20260925 10:14:02.098344 14193 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:02.098553 14193 heartbeater.cc:511] Master 127.3.152.254:45019 requested a full tablet report, sending...
I20260925 10:14:02.099138 14002 ts_manager.cc:194] Registered new tserver with Master: 1d3ff9bd036c40fe8c03ef23da4bd058 (127.3.152.193:41617)
I20260925 10:14:02.099828 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:54481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:14:02.117249 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:02.199951 14198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:02.200181 14198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:02.200204 14198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:02.201287 14198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:02.201325 14198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:02.201351 14198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:14:02.202836 14198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:02.203040 14198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:02.203215 14198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:02.208154 14203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:02.208176 14206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:02.208173 14204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:02.208683 14198 server_base.cc:1034] running on GCE node
I20260925 10:14:02.208896 14198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:02.209095 14198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:02.210261 14198 hybrid_clock.cc:648] HybridClock initialized: now 1790331242210242 us; error 30 us; skew 500 ppm
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:02.212942 14198 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:14:02.213410 14198 webserver.cc:460] Webserver started at http://127.3.152.194:44001/ using document root <none> and password file <none>
I20260925 10:14:02.213665 14198 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:02.213739 14198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:02.213891 14198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:02.214967 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "39db94dc47984ce79d002786ebc68a59"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "117103c44765c5997088d28d6a35de1f"
server_key_iv: "d521e229466a82d106caf930c702989e"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.215338 14198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "39db94dc47984ce79d002786ebc68a59"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "117103c44765c5997088d28d6a35de1f"
server_key_iv: "d521e229466a82d106caf930c702989e"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.216784 14198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:14:02.217614 14213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.217865 14198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:14:02.217973 14198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "39db94dc47984ce79d002786ebc68a59"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "117103c44765c5997088d28d6a35de1f"
server_key_iv: "d521e229466a82d106caf930c702989e"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.218056 14198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:02.227035 14198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:02.229143 14198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:02.229317 14198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:02.229562 14198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:02.229912 14198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:02.229969 14198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.230008 14198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:02.230038 14198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.236121 14198 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:46313
I20260925 10:14:02.236173 14326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:46313 every 8 connection(s)
I20260925 10:14:02.236511 14198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:14:02.242578 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 14198
I20260925 10:14:02.242677 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:02.242920 3683 external_mini_cluster.cc:1428] Setting key 3b5b29ee6d4fefb35aa2f8a7401ff435
I20260925 10:14:02.247067 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.237307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:57525 (local address 127.3.152.254:45019)
0925 10:14:02.237383 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.237385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.237652 (+ 267us) server_negotiation.cc:407] Connection header received
0925 10:14:02.238118 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.238119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.238132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.238200 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:02.239140 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.239644 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.240787 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.240986 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.241055 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.243459 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:02.243461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:02.243461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:02.243468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:02.245466 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.246035 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.246036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.246036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.246094 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.246424 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.246424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.246424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.246535 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:02.246589 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.246926 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.246999 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:14:02.247416 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.237293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:45019 (local address 127.3.152.194:57525)
0925 10:14:02.237459 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.237504 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.237987 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.238256 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.238260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.238839 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:02.239044 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.239049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.239752 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.239753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.240682 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.240685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.241075 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.241076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.241186 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.241725 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:02.241733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:02.243370 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:02.245564 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.245572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.245575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.245935 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.246168 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.246168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.246169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.246323 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.246604 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:02.246610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:02.246787 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.246907 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.247123 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260925 10:14:02.247987 14327 heartbeater.cc:347] Connected to a master server at 127.3.152.254:45019
I20260925 10:14:02.248111 14327 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:02.248332 14327 heartbeater.cc:511] Master 127.3.152.254:45019 requested a full tablet report, sending...
I20260925 10:14:02.248751 14002 ts_manager.cc:194] Registered new tserver with Master: 39db94dc47984ce79d002786ebc68a59 (127.3.152.194:46313)
I20260925 10:14:02.249141 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:57525
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:14:02.271299 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:02.361192 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:02.361428 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:02.361454 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:02.362762 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:02.362811 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:02.362841 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:14:02.364558 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:02.364830 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:02.365131 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:02.370582 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:02.370625 14332 server_base.cc:1034] running on GCE node
W20260925 10:14:02.370591 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:02.370738 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:02.370998 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:02.371275 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:02.372449 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790331242372408 us; error 66 us; skew 500 ppm
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:02.375449 14332 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:14:02.375855 14332 webserver.cc:460] Webserver started at http://127.3.152.195:43881/ using document root <none> and password file <none>
I20260925 10:14:02.376137 14332 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:02.376209 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:02.376369 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:14:02.377470 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "500c3ded629c4bdda55621e002b88b46"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "03504bfc6123348f89f44f86a66069d6"
server_key_iv: "3d438f954961979c83e12e40d5a53608"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.377842 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "500c3ded629c4bdda55621e002b88b46"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "03504bfc6123348f89f44f86a66069d6"
server_key_iv: "3d438f954961979c83e12e40d5a53608"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.379194 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:14:02.380095 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.380375 14332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:02.380481 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "500c3ded629c4bdda55621e002b88b46"
format_stamp: "Formatted at 2026-09-25 10:14:02 on dist-test-slave-dzwq"
server_key: "03504bfc6123348f89f44f86a66069d6"
server_key_iv: "3d438f954961979c83e12e40d5a53608"
server_key_version: "encryptionkey@0"
I20260925 10:14:02.380558 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:02.396512 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:02.398708 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:02.398893 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:02.399174 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:02.399533 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:02.399590 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.399633 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:02.399660 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:02.405926 14332 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:33107
I20260925 10:14:02.405983 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:33107 every 8 connection(s)
I20260925 10:14:02.406343 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:14:02.406702 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 14332
I20260925 10:14:02.406793 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:14:02.406991 3683 external_mini_cluster.cc:1428] Setting key 297a61d64b091ea5a3de65ac8c4a43fc
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[13957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331242, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:02.416368 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.407359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:49713 (local address 127.3.152.254:45019)
0925 10:14:02.407410 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.407413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.407422 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:14:02.408135 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.408136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.408146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.408193 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:02.409152 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.409661 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.410711 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.410902 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.410957 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.413364 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:02.413366 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:02.413366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:02.413375 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:02.414857 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.415296 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.415297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.415298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.415363 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.415626 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.415627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.415627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.415757 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:02.415836 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.416183 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.416270 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:14:02.416569 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.407117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:45019 (local address 127.3.152.195:49713)
0925 10:14:02.407292 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.407337 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.408024 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.408256 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.408259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.408834 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:02.409050 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.409055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.409751 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.409751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.410578 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.410580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.410983 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.410984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.411110 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.411667 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:02.411675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:02.413257 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:02.414963 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.414970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.414973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.415216 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.415442 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:02.415443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:02.415443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:02.415557 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:02.415827 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:02.415834 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:02.416030 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.416161 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.416326 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260925 10:14:02.417124 14461 heartbeater.cc:347] Connected to a master server at 127.3.152.254:45019
I20260925 10:14:02.417224 14461 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:02.417382 14461 heartbeater.cc:511] Master 127.3.152.254:45019 requested a full tablet report, sending...
I20260925 10:14:02.417735 14002 ts_manager.cc:194] Registered new tserver with Master: 500c3ded629c4bdda55621e002b88b46 (127.3.152.195:33107)
I20260925 10:14:02.418179 14002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:49713
I20260925 10:14:02.418872 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:14:02.426635 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.420064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.3.152.254:45019)
0925 10:14:02.420130 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.420132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.420284 (+ 152us) server_negotiation.cc:407] Connection header received
0925 10:14:02.420320 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.420320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.420329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.420377 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:02.420937 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.421434 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.422656 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.422872 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.422927 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.424182 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:02.424183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:02.424183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:02.424189 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:02.425403 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.425762 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.425763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.425763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.425825 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:02.426047 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:02.426048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:02.426049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:02.426205 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:02.426330 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.426444 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.426528 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:14:02.428481 14002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:14:02.430146 14002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:14:02.436796 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:41617 (local address 127.0.0.1:39482)
0925 10:14:02.432998 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.433006 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.433058 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.433289 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.433292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.433295 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:02.433376 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.433378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.434402 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.434403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.435592 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.435595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.436455 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.436456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.436540 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.436555 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.436647 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.436700 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":151,"threads_started":1}
I20260925 10:14:02.436807 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.3.152.193:41617)
0925 10:14:02.432919 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.432925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.433073 (+ 148us) server_negotiation.cc:407] Connection header received
0925 10:14:02.433120 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.433123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.433149 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.433500 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:02.433608 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.434289 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.435679 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.436369 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.436447 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.436505 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.436669 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.436722 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260925 10:14:02.436990 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46313 (local address 127.0.0.1:40850)
0925 10:14:02.432918 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.432962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.433035 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.433380 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.433381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.433383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:02.433449 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.433450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.434831 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.434832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.435852 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.435853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.436760 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.436761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.436834 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.436839 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.436871 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.436921 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260925 10:14:02.437456 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.3.152.194:46313)
0925 10:14:02.433201 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.433206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.433216 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:14:02.433256 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.433258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.433276 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.433524 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:02.433610 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.434722 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.435927 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.436663 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.437227 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.437302 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.437323 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.437356 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":76,"threads_started":1}
I20260925 10:14:02.437896 14127 tablet_service.cc:1467] Processing CreateTablet for tablet ff45009e79934bd78604cd2cf86d5da5 (DEFAULT_TABLE table=test-table [id=ab6db5dd88d747a0a618c74e5068275c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:02.438293 14127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45009e79934bd78604cd2cf86d5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:02.438323 14261 tablet_service.cc:1467] Processing CreateTablet for tablet ff45009e79934bd78604cd2cf86d5da5 (DEFAULT_TABLE table=test-table [id=ab6db5dd88d747a0a618c74e5068275c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:02.438584 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:33107 (local address 127.0.0.1:44810)
0925 10:14:02.433168 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.433177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.433216 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.433497 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.433498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.433499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:02.433537 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.433538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.435921 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.435922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.436813 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.436816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.438372 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.438373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.438436 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.438441 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.438472 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.438513 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":68,"threads_started":1}
I20260925 10:14:02.438643 14261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45009e79934bd78604cd2cf86d5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:02.438978 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.432981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.3.152.195:33107)
0925 10:14:02.433336 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.433340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.433350 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:14:02.433385 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.433387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.433401 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.433606 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:02.433674 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.434716 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.437311 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.438259 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.438781 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.438846 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.438873 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.438916 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":99,"threads_started":1}
I20260925 10:14:02.439534 14395 tablet_service.cc:1467] Processing CreateTablet for tablet ff45009e79934bd78604cd2cf86d5da5 (DEFAULT_TABLE table=test-table [id=ab6db5dd88d747a0a618c74e5068275c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:02.439862 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff45009e79934bd78604cd2cf86d5da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:02.441992 14478 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Bootstrap starting.
I20260925 10:14:02.442121 14479 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Bootstrap starting.
I20260925 10:14:02.442740 14479 tablet_bootstrap.cc:654] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:02.442813 14480 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Bootstrap starting.
I20260925 10:14:02.442760 14478 tablet_bootstrap.cc:654] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:02.443324 14479 log.cc:824] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:02.443383 14478 log.cc:824] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:02.443459 14480 tablet_bootstrap.cc:654] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:02.443991 14480 log.cc:824] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:02.444145 14479 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: No bootstrap required, opened a new log
I20260925 10:14:02.444216 14479 ts_tablet_manager.cc:1397] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:14:02.445554 14479 raft_consensus.cc:348] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.445710 14479 raft_consensus.cc:374] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:02.445735 14479 raft_consensus.cc:729] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39db94dc47984ce79d002786ebc68a59, State: Initialized, Role: FOLLOWER
I20260925 10:14:02.445869 14479 consensus_queue.cc:260] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.446151 14479 ts_tablet_manager.cc:1428] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:14:02.446261 14327 heartbeater.cc:503] Master 127.3.152.254:45019 was elected leader, sending a full tablet report...
I20260925 10:14:02.446889 14478 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: No bootstrap required, opened a new log
I20260925 10:14:02.446964 14478 ts_tablet_manager.cc:1397] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:14:02.446965 14480 tablet_bootstrap.cc:492] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: No bootstrap required, opened a new log
I20260925 10:14:02.447032 14480 ts_tablet_manager.cc:1397] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 10:14:02.448675 14480 raft_consensus.cc:348] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.448830 14480 raft_consensus.cc:374] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:02.448874 14480 raft_consensus.cc:729] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500c3ded629c4bdda55621e002b88b46, State: Initialized, Role: FOLLOWER
I20260925 10:14:02.448992 14480 consensus_queue.cc:260] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.448945 14478 raft_consensus.cc:348] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.449079 14478 raft_consensus.cc:374] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:02.449131 14478 raft_consensus.cc:729] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3ff9bd036c40fe8c03ef23da4bd058, State: Initialized, Role: FOLLOWER
I20260925 10:14:02.449293 14480 ts_tablet_manager.cc:1428] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:14:02.449254 14478 consensus_queue.cc:260] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.449342 14461 heartbeater.cc:503] Master 127.3.152.254:45019 was elected leader, sending a full tablet report...
I20260925 10:14:02.449527 14478 ts_tablet_manager.cc:1428] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:14:02.449678 14193 heartbeater.cc:503] Master 127.3.152.254:45019 was elected leader, sending a full tablet report...
W20260925 10:14:02.487133 14328 tablet.cc:2369] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:14:02.559917 14486 raft_consensus.cc:482] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:14:02.560108 14486 raft_consensus.cc:504] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.560480 14486 leader_election.cc:288] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39db94dc47984ce79d002786ebc68a59 (127.3.152.194:46313), 500c3ded629c4bdda55621e002b88b46 (127.3.152.195:33107)
I20260925 10:14:02.563618 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.560811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:55037 (local address 127.3.152.194:46313)
0925 10:14:02.560882 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.560884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.560895 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:14:02.560923 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.560924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.560934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.560989 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:02.561192 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.561735 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.562789 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.563256 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.563339 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.563348 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.563498 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.563539 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:14:02.563618 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.560697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46313 (local address 127.3.152.193:55037)
0925 10:14:02.560769 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.560786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.560853 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.561001 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.561002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.561004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:02.561109 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.561111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.561840 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.561841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.562695 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.562698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.563347 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.563348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.563424 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.563428 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.563493 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.563536 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":11}
I20260925 10:14:02.563720 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.560910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:47835 (local address 127.3.152.195:33107)
0925 10:14:02.560961 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.560963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.561094 (+ 131us) server_negotiation.cc:407] Connection header received
0925 10:14:02.561213 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.561215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.561224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.561312 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:02.561478 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.561991 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.562945 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.563337 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.563424 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.563432 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.563588 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.563629 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:14:02.564003 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.560847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:33107 (local address 127.3.152.193:47835)
0925 10:14:02.561009 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.561024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:02.561108 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:02.561322 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:02.561323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:02.561325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:02.561403 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.561406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.562080 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.562081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.562830 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:02.562832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.563424 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.563424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:02.563502 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.563505 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:14:02.563580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:14:02.563624 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260925 10:14:02.564014 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45009e79934bd78604cd2cf86d5da5" candidate_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39db94dc47984ce79d002786ebc68a59" is_pre_election: true
I20260925 10:14:02.564236 14281 raft_consensus.cc:2454] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3ff9bd036c40fe8c03ef23da4bd058 in term 0.
I20260925 10:14:02.564352 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45009e79934bd78604cd2cf86d5da5" candidate_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500c3ded629c4bdda55621e002b88b46" is_pre_election: true
I20260925 10:14:02.564527 14415 raft_consensus.cc:2454] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d3ff9bd036c40fe8c03ef23da4bd058 in term 0.
I20260925 10:14:02.564514 14081 leader_election.cc:302] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3ff9bd036c40fe8c03ef23da4bd058, 39db94dc47984ce79d002786ebc68a59; no voters:
I20260925 10:14:02.564688 14486 raft_consensus.cc:2781] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:14:02.564743 14486 raft_consensus.cc:482] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:14:02.564769 14486 raft_consensus.cc:3037] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:02.565528 14486 raft_consensus.cc:504] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.565665 14486 leader_election.cc:288] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [CANDIDATE]: Term 1 election: Requested vote from peers 39db94dc47984ce79d002786ebc68a59 (127.3.152.194:46313), 500c3ded629c4bdda55621e002b88b46 (127.3.152.195:33107)
I20260925 10:14:02.565845 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45009e79934bd78604cd2cf86d5da5" candidate_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "500c3ded629c4bdda55621e002b88b46"
I20260925 10:14:02.565867 14281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff45009e79934bd78604cd2cf86d5da5" candidate_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39db94dc47984ce79d002786ebc68a59"
I20260925 10:14:02.565943 14415 raft_consensus.cc:3037] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:02.565964 14281 raft_consensus.cc:3037] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:02.566664 14281 raft_consensus.cc:2454] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3ff9bd036c40fe8c03ef23da4bd058 in term 1.
I20260925 10:14:02.566841 14415 raft_consensus.cc:2454] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3ff9bd036c40fe8c03ef23da4bd058 in term 1.
I20260925 10:14:02.566838 14081 leader_election.cc:302] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3ff9bd036c40fe8c03ef23da4bd058, 39db94dc47984ce79d002786ebc68a59; no voters:
I20260925 10:14:02.566926 14486 raft_consensus.cc:2781] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:14:02.567063 14486 raft_consensus.cc:686] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3ff9bd036c40fe8c03ef23da4bd058, State: Running, Role: LEADER
I20260925 10:14:02.567195 14486 consensus_queue.cc:237] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } }
I20260925 10:14:02.568135 14001 catalog_manager.cc:5495] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d3ff9bd036c40fe8c03ef23da4bd058 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3ff9bd036c40fe8c03ef23da4bd058" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 41617 } health_report { overall_health: HEALTHY } } }
I20260925 10:14:02.583449 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.580554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39492 (local address 127.3.152.193:41617)
0925 10:14:02.580613 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:02.580616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:02.580625 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:14:02.580687 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:02.580688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:02.580709 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:02.580800 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:14:02.580958 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.581448 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.582472 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:02.582788 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:02.582851 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:02.583005 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:14:02.583164 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:14:02.583295 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:02.583312 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 10:14:02.583335 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 10:14:02.585503 14415 raft_consensus.cc:1261] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 1 FOLLOWER]: Refusing update from remote peer 1d3ff9bd036c40fe8c03ef23da4bd058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:14:02.585503 14281 raft_consensus.cc:1261] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 1 FOLLOWER]: Refusing update from remote peer 1d3ff9bd036c40fe8c03ef23da4bd058: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:14:02.585914 14486 consensus_queue.cc:1035] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "500c3ded629c4bdda55621e002b88b46" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:14:02.586081 14488 consensus_queue.cc:1035] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39db94dc47984ce79d002786ebc68a59" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 10:14:02.587006 14194 tablet.cc:2369] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:14:02.591096 14001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:14:02.591197 14001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:14:02.592013 14001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a55649ce90b43f2b755697531d8ee65: Sending DeleteTablet for 3 replicas of tablet ff45009e79934bd78604cd2cf86d5da5
I20260925 10:14:02.592388 14261 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45009e79934bd78604cd2cf86d5da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:02 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40850
I20260925 10:14:02.592388 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45009e79934bd78604cd2cf86d5da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:02 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:44810
I20260925 10:14:02.592613 14127 tablet_service.cc:1514] Processing DeleteTablet for tablet ff45009e79934bd78604cd2cf86d5da5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:02 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:39482
I20260925 10:14:02.592656 14502 tablet_replica.cc:330] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: stopping tablet replica
I20260925 10:14:02.592757 14502 raft_consensus.cc:2229] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:14:02.592849 14503 tablet_replica.cc:330] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058: stopping tablet replica
I20260925 10:14:02.592865 14502 raft_consensus.cc:2258] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:14:02.593029 14503 raft_consensus.cc:2229] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:14:02.593108 14501 tablet_replica.cc:330] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: stopping tablet replica
I20260925 10:14:02.593170 14503 raft_consensus.cc:2258] T ff45009e79934bd78604cd2cf86d5da5 P 1d3ff9bd036c40fe8c03ef23da4bd058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:14:02.593246 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 14064
I20260925 10:14:02.593268 14501 raft_consensus.cc:2229] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:14:02.595484 14502 ts_tablet_manager.cc:1905] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:14:02.596494 14501 raft_consensus.cc:2258] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:14:02.597018 14501 ts_tablet_manager.cc:1905] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:14:02.597394 14502 ts_tablet_manager.cc:1918] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:14:02.597472 14502 log.cc:1196] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/ff45009e79934bd78604cd2cf86d5da5
I20260925 10:14:02.597735 14502 ts_tablet_manager.cc:1939] T ff45009e79934bd78604cd2cf86d5da5 P 500c3ded629c4bdda55621e002b88b46: Deleting consensus metadata
I20260925 10:14:02.598210 13987 catalog_manager.cc:4841] TS 500c3ded629c4bdda55621e002b88b46 (127.3.152.195:33107): tablet ff45009e79934bd78604cd2cf86d5da5 (table test-table [id=ab6db5dd88d747a0a618c74e5068275c]) successfully deleted
W20260925 10:14:02.598613 13988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:41617 (error 108)
I20260925 10:14:02.599164 14501 ts_tablet_manager.cc:1918] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:14:02.599228 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.599093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:41617 (local address 127.0.0.1:39498)
0925 10:14:02.599149 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.599192 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:41617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:14:02.599244 14501 log.cc:1196] T ff45009e79934bd78604cd2cf86d5da5 P 39db94dc47984ce79d002786ebc68a59: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/ff45009e79934bd78604cd2cf86d5da5
I20260925 10:14:02.599313 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 14198
W20260925 10:14:02.599349 13988 catalog_manager.cc:4584] TS 1d3ff9bd036c40fe8c03ef23da4bd058 (127.3.152.193:41617): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff45009e79934bd78604cd2cf86d5da5: Network error: Client connection negotiation failed: client connection to 127.3.152.193:41617: connect: Connection refused (error 111)
I20260925 10:14:02.604228 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:02.604103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46313 (local address 127.0.0.1:40858)
0925 10:14:02.604154 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:02.604180 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.194:46313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:14:02.604722 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 14332
I20260925 10:14:02.609309 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 13970
2026-09-25T10:14:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2066 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[14507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[14507](info): set up 2 sockets
Sep 25 10:14:02 dist-test-slave-dzwq krb5kdc[14507](info): commencing operation
krb5kdc: starting...
W20260925 10:14:03.669668 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:14:03 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:14:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangeradmin_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangeradmin_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangerlookup_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangerlookup_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
I20260925 10:14:03.855397 3683 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcY4CcD/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/postgres -l logfile start
2026-09-25 10:14:06.653 UTC [14537] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:14:06.654 UTC [14537] LOG: listening on IPv4 address "127.3.152.212", port 44571
2026-09-25 10:14:06.658 UTC [14537] LOG: listening on Unix socket "/tmp/.s.PGSQL.44571"
2026-09-25 10:14:06.698 UTC [14540] LOG: database system was shut down at 2026-09-25 10:14:04 UTC
2026-09-25 10:14:06.702 UTC [14537] LOG: database system is ready to accept connections
I20260925 10:14:07.700886 3683 mini_postgres.cc:96] Postgres bound to 44571
2026-09-25 10:14:07.707 UTC [14548] FATAL: database "slave" does not exist
127.3.152.212:44571 - accepting connections
I20260925 10:14:07.708107 3683 mini_ranger.cc:161] Starting Ranger...
I20260925 10:14:07.724633 3683 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:14:08.282326 3683 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:14:08.282426 3683 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin
2026-09-25 10:14:08,603 [I] DB FLAVOR :POSTGRES
2026-09-25 10:14:08,603 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:14:08,603 [I] Checking connection..
2026-09-25 10:14:08,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:14:08,882 [I] Checking connection passed.
2026-09-25 10:14:08,882 [I] --------- Verifying version history table ---------
2026-09-25 10:14:08,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:09,138 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:14:09,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:09,384 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:14:09,384 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:14:09,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:14:09.643 UTC [14612] WARNING: there is no transaction in progress
2026-09-25 10:14:09,650 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:14:09,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:09,913 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:14:09,913 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:14:09,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:14:10,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:14:10,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dzwq.c.gcp-upstream.internal','N') ;"
2026-09-25 10:14:10,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:14:10,897 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:14:10,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:14:11,162 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:14:11,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:14:11,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:14:11,646 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:14:11,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:14:12.676 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.693 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.710 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.726 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.744 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.767 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.829 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.836 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.853 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.866 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.877 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.891 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.902 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.909 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.916 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:12.923 UTC [14720] WARNING: there is no transaction in progress
2026-09-25 10:14:13,644 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:14:13,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:14:13,883 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:14:13,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:14:14,139 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:14:14,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:14:14,399 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:14:14,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:14:14,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:14:14,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dzwq.c.gcp-upstream.internal';"
2026-09-25 10:14:15,146 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:14:15,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:14:15,406 [I] DB_PATCHES have already been applied
I20260925 10:14:15.412204 3683 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcY4CcD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:14:15.412281 3683 mini_ranger.cc:193] Using host: 127.3.152.212
I20260925 10:14:15.415360 3683 mini_ranger.cc:227] Ranger admin URL: http://127.3.152.212:52661
Sep 25, 2026 10:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:14:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:14:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:14:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:14:16.557837 3683 mini_ranger.cc:160] Time spent starting Ranger: real 8.850s user 0.000s sys 0.010s
Sep 25, 2026 10:14:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:14:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 10:14:36 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:14:36 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:14:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:14:38.231863 3683 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangerkms_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/rangerkms_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.152.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.152.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:14:38 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:14:38.328411 3683 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:14:38.342000 3683 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:14:39.274507 3683 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:14:39.274595 3683 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms
I20260925 10:14:39.274626 3683 mini_ranger_kms.cc:227] Using postgres at 127.3.152.212:44571
2026-09-25 10:14:39,711 [I] DB FLAVOR :POSTGRES
2026-09-25 10:14:39,711 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:14:39,712 [I] Checking connection
2026-09-25 10:14:39,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:14:39,954 [I] connection success
2026-09-25 10:14:39,954 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:14:39,954 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:14:39,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:14:40,225 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:14:40,225 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:14:40,226 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:14:40,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:44571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:14:40,519 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:14:40.997547 3683 mini_ranger_kms.cc:310] Created kms service
I20260925 10:14:41.222971 3683 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:14:41.272791 3683 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:14:41 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331278, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:41.716666 3683 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:14:41.716792 3683 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:14:41.716814 3683 mini_ranger_kms.cc:240] Using host: 127.3.152.212
I20260925 10:14:41.720062 3683 mini_ranger_kms.cc:276] Ranger KMS PID: 14900
I20260925 10:14:41.720211 3683 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.152.212:40213
Sep 25, 2026 10:14:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:14:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:14:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:14:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:14:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:14:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:14:43.940301 3683 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.612s user 0.007s sys 0.187s
I20260925 10:14:43.940421 3683 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:14:43.940536 3683 mini_ranger_kms.cc:401] 127.3.152.212:40213/kms/v1/keys
Sep 25 10:14:44 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:14:44 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
Sep 25 10:14:45 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
Sep 25 10:14:47 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:14:47.447290 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37277
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:37277
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:40213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:47.529930 14951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:47.530189 14951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:47.530241 14951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:47.531415 14951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:14:47.531487 14951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:47.531512 14951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:47.531539 14951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:14:47.531569 14951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:14:47.533226 14951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.152.212:40213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:37277
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:47.533519 14951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:47.533767 14951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:47.540521 14957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:47.540614 14951 server_base.cc:1034] running on GCE node
W20260925 10:14:47.540619 14959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:47.540524 14956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:47.541158 14951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:47.541503 14951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:47.542689 14951 hybrid_clock.cc:648] HybridClock initialized: now 1790331287542666 us; error 42 us; skew 500 ppm
Sep 25 10:14:47 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331287, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:47.553757 14951 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:14:47.554464 14951 webserver.cc:460] Webserver started at http://127.3.152.254:41483/ using document root <none> and password file <none>
I20260925 10:14:47.554791 14951 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:47.554863 14951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:47.555102 14951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:14:47 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331287, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:47.616860 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "4b160ad6a39f4bc38a4b2a6609516914"
format_stamp: "Formatted at 2026-09-25 10:14:47 on dist-test-slave-dzwq"
server_key: "ee39eea855cdef887929b6896bf5acda"
server_key_iv: "f2d0ee47ec516e501b127199da852de0"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:47.617393 14951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "4b160ad6a39f4bc38a4b2a6609516914"
format_stamp: "Formatted at 2026-09-25 10:14:47 on dist-test-slave-dzwq"
server_key: "ee39eea855cdef887929b6896bf5acda"
server_key_iv: "f2d0ee47ec516e501b127199da852de0"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:47.619007 14951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:14:47.634713 14966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:47.634971 14951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:47.635087 14951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "4b160ad6a39f4bc38a4b2a6609516914"
format_stamp: "Formatted at 2026-09-25 10:14:47 on dist-test-slave-dzwq"
server_key: "ee39eea855cdef887929b6896bf5acda"
server_key_iv: "f2d0ee47ec516e501b127199da852de0"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:47.635195 14951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:47.671800 14951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:47.673895 14951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:47.674054 14951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:47.679533 14951 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:37277
I20260925 10:14:47.679617 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:37277 every 8 connection(s)
I20260925 10:14:47.680034 14951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:14:47.680783 15019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:47.683069 15019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: Bootstrap starting.
I20260925 10:14:47.683689 15019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:47.684069 15019 log.cc:824] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:47.684604 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 14951
I20260925 10:14:47.684711 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:14:47.684888 15019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: No bootstrap required, opened a new log
I20260925 10:14:47.684926 3683 external_mini_cluster.cc:1428] Setting key c413c4827fe7c5a253039ca341df86f0
I20260925 10:14:47.687952 15019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } }
I20260925 10:14:47.688191 15019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:47.688237 15019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b160ad6a39f4bc38a4b2a6609516914, State: Initialized, Role: FOLLOWER
I20260925 10:14:47.688383 15019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } }
I20260925 10:14:47.688460 15019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:14:47.688488 15019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:14:47.688531 15019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:47.689216 15019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } }
I20260925 10:14:47.689347 15019 leader_election.cc:302] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b160ad6a39f4bc38a4b2a6609516914; no voters:
I20260925 10:14:47.689527 15019 leader_election.cc:288] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:14:47.689633 15024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:14:47.689806 15024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [term 1 LEADER]: Becoming Leader. State: Replica: 4b160ad6a39f4bc38a4b2a6609516914, State: Running, Role: LEADER
I20260925 10:14:47.689910 15024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } }
I20260925 10:14:47.690229 15024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b160ad6a39f4bc38a4b2a6609516914. Latest consensus state: current_term: 1 leader_uuid: "4b160ad6a39f4bc38a4b2a6609516914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } } }
I20260925 10:14:47.690295 15024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:47.689797 15019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:14:47.691390 15026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b160ad6a39f4bc38a4b2a6609516914" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b160ad6a39f4bc38a4b2a6609516914" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37277 } } }
I20260925 10:14:47.691552 15026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914 [sys.catalog]: This master's current role is: LEADER
I20260925 10:14:47.691725 15028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:14:47.691846 15019 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:14:47.692396 15028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:14:47 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:47.699069 15028 catalog_manager.cc:1347] Generated new cluster ID: d172802970334c52990877143b1dd2f4
I20260925 10:14:47.699168 15028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:14:47.700007 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:47.685495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.3.152.254:37277)
0925 10:14:47.685666 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:47.685676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:47.685722 (+ 46us) server_negotiation.cc:407] Connection header received
0925 10:14:47.686551 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:47.686562 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:47.686631 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:47.686797 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:47.687580 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:47.689067 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:47.690118 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:47.690769 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:47.690915 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:47.694242 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:47.694303 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:47.694304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:47.694430 (+ 126us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:47.697603 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:47.698059 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:47.698062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:47.698062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:47.698175 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:47.698407 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:47.698407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:47.698408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:47.698850 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:47.699024 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:47.699515 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 10:14:47.699689 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260925 10:14:47.711939 15028 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:14:47.712810 15028 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:14:47.726006 15028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: Generated new TSK 0
I20260925 10:14:47.726316 15028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:14:48.415417 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:40213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:48.539139 15064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:48.539434 15064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:48.539527 15064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:48.541359 15064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:48.541430 15064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:48.541472 15064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:14:48.543713 15064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.152.212:40213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:48.544261 15064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:48.544651 15064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:48.551532 15070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:48.551838 15064 server_base.cc:1034] running on GCE node
W20260925 10:14:48.552114 15072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:48.552666 15069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:48.552989 15064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:48.553288 15064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:48.556587 15064 hybrid_clock.cc:648] HybridClock initialized: now 1790331288556540 us; error 45 us; skew 500 ppm
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:48.560263 15064 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:14:48.560878 15064 webserver.cc:460] Webserver started at http://127.3.152.193:37343/ using document root <none> and password file <none>
I20260925 10:14:48.561168 15064 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:48.561268 15064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:48.561440 15064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:48.578161 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "c4c4298a914a45adbfa884b71cb7e997"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "4e45d780c31458b1d73af96606be5b2c"
server_key_iv: "fbe2e11905fc5343356f582fc142a635"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.578655 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "c4c4298a914a45adbfa884b71cb7e997"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "4e45d780c31458b1d73af96606be5b2c"
server_key_iv: "fbe2e11905fc5343356f582fc142a635"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.584489 15064 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260925 10:14:48.598749 15079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.598949 15064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:14:48.599063 15064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "c4c4298a914a45adbfa884b71cb7e997"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "4e45d780c31458b1d73af96606be5b2c"
server_key_iv: "fbe2e11905fc5343356f582fc142a635"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.599192 15064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:48.627800 15064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:48.630175 15064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:48.630342 15064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:48.630577 15064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:48.630977 15064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:48.631036 15064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.631069 15064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:48.631090 15064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.638566 15064 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:42083
I20260925 10:14:48.638753 15194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:42083 every 8 connection(s)
I20260925 10:14:48.639089 15064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:14:48.646903 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 15064
I20260925 10:14:48.647025 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:48.647224 3683 external_mini_cluster.cc:1428] Setting key 646ffdaae93e729bfd10d34c2c947106
I20260925 10:14:48.651530 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:48.640036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37277 (local address 127.3.152.193:44775)
0925 10:14:48.640342 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:48.640394 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:48.641149 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:48.641471 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:48.641476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:48.641743 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:48.642100 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:48.642108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.643183 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.643184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:48.644540 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:48.644543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.645053 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.645054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:48.645218 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:48.645802 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:48.645814 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:48.647566 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:48.649736 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:48.649745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:48.649748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:48.650075 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:48.650326 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:48.650327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:48.650327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:48.650462 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:48.650759 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:48.650766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:48.650946 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 10:14:48.651067 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 10:14:48.651260 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260925 10:14:48.652189 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:48.640058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:44775 (local address 127.3.152.254:37277)
0925 10:14:48.640265 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:48.640268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:48.640521 (+ 253us) server_negotiation.cc:407] Connection header received
0925 10:14:48.641281 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:48.641282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:48.641298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:48.641385 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:48.642225 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.643070 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.644642 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.644953 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.645017 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:48.647709 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:48.647710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:48.647710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:48.647717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:48.649633 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:48.650197 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:48.650198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:48.650198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:48.650265 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:48.650524 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:48.650525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:48.650526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:48.650665 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:48.650730 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:48.651929 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0925 10:14:48.652061 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260925 10:14:48.652603 15195 heartbeater.cc:347] Connected to a master server at 127.3.152.254:37277
I20260925 10:14:48.652715 15195 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:48.652918 15195 heartbeater.cc:511] Master 127.3.152.254:37277 requested a full tablet report, sending...
I20260925 10:14:48.653548 14983 ts_manager.cc:194] Registered new tserver with Master: c4c4298a914a45adbfa884b71cb7e997 (127.3.152.193:42083)
I20260925 10:14:48.654508 14983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:44775
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:14:48.684974 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:40213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 10:14:48.805543 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:48.805827 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:48.805867 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:48.807725 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:48.807843 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:48.807915 15201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:14:48.810314 15201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.152.212:40213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:48.810760 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:48.811076 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:48.821281 15211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:48.821545 15209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:48.821918 15208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:48.823159 15201 server_base.cc:1034] running on GCE node
I20260925 10:14:48.823555 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:48.823884 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:48.825088 15201 hybrid_clock.cc:648] HybridClock initialized: now 1790331288825053 us; error 52 us; skew 500 ppm
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:48.828450 15201 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:14:48.829043 15201 webserver.cc:460] Webserver started at http://127.3.152.194:44335/ using document root <none> and password file <none>
I20260925 10:14:48.829381 15201 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:48.829497 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:48.829691 15201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:48.852384 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "667b5b13d5c3433aa26ccf0d9481b34c"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "c74db73d39346a96f7112f86d4b186ab"
server_key_iv: "f04e2199f6871ff62fe09de954134393"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.852814 15201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "667b5b13d5c3433aa26ccf0d9481b34c"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "c74db73d39346a96f7112f86d4b186ab"
server_key_iv: "f04e2199f6871ff62fe09de954134393"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.857084 15201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 10:14:48.875826 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.876155 15201 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 10:14:48.876289 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "667b5b13d5c3433aa26ccf0d9481b34c"
format_stamp: "Formatted at 2026-09-25 10:14:48 on dist-test-slave-dzwq"
server_key: "c74db73d39346a96f7112f86d4b186ab"
server_key_iv: "f04e2199f6871ff62fe09de954134393"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:48.876410 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:48.896062 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:48.898661 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:48.898813 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:48.899081 15201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:48.899482 15201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:48.899538 15201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.899572 15201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:48.899596 15201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:48.906510 15201 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:42959
I20260925 10:14:48.906975 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:14:48.912334 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 15201
I20260925 10:14:48.912460 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:14:48.912747 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:42959 every 8 connection(s)
I20260925 10:14:48.912688 3683 external_mini_cluster.cc:1428] Setting key ed679d17131e40bcdd3b05acfe9bac81
Sep 25 10:14:48 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:48.918167 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:48.908311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:35667 (local address 127.3.152.254:37277)
0925 10:14:48.908586 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:48.908589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:48.908629 (+ 40us) server_negotiation.cc:407] Connection header received
0925 10:14:48.909208 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:48.909209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:48.909229 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:48.909276 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:48.909873 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.910636 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.911800 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.912141 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.912214 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:48.914692 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:48.914693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:48.914694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:48.914703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:48.916415 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:48.916937 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:48.916938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:48.916939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:48.917000 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:48.917352 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:48.917353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:48.917354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:48.917503 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:48.917576 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:48.917952 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 10:14:48.918051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260925 10:14:48.918399 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:48.908319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37277 (local address 127.3.152.194:35667)
0925 10:14:48.908506 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:48.908551 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:48.909081 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:48.909368 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:48.909371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:48.909564 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:48.909765 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:48.909769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.910752 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.910753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:48.911688 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:48.911690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:48.912292 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:48.912293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:48.912472 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:48.913107 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:48.913116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:48.914584 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:48.916498 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:48.916506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:48.916509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:48.916837 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:48.917075 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:48.917076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:48.917076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:48.917242 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:48.917587 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:48.917595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:48.917780 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 10:14:48.917905 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 10:14:48.918083 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260925 10:14:48.919037 15333 heartbeater.cc:347] Connected to a master server at 127.3.152.254:37277
I20260925 10:14:48.919169 15333 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:48.919387 15333 heartbeater.cc:511] Master 127.3.152.254:37277 requested a full tablet report, sending...
I20260925 10:14:48.919878 14983 ts_manager.cc:194] Registered new tserver with Master: 667b5b13d5c3433aa26ccf0d9481b34c (127.3.152.194:42959)
I20260925 10:14:48.920466 14983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:35667
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:14:48.941170 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:40213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:14:49.035810 15338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:14:49.036113 15338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:14:49.036154 15338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:14:49.037914 15338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:14:49.037973 15338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:14:49.038024 15338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:14:49.040491 15338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.152.212:40213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:14:49.040776 15338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:14:49.041057 15338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:14:49.047117 15346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:49.048206 15338 server_base.cc:1034] running on GCE node
W20260925 10:14:49.048445 15343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:14:49.048743 15344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:14:49.049043 15338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:14:49.049331 15338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:14:49.050482 15338 hybrid_clock.cc:648] HybridClock initialized: now 1790331289050460 us; error 38 us; skew 500 ppm
Sep 25 10:14:49 dist-test-slave-dzwq krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:14:49.053562 15338 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:14:49.054113 15338 webserver.cc:460] Webserver started at http://127.3.152.195:45433/ using document root <none> and password file <none>
I20260925 10:14:49.054404 15338 fs_manager.cc:359] Metadata directory not provided
I20260925 10:14:49.054478 15338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:14:49.054628 15338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:14:49 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:49.068346 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "bb18727057dc485db0938ae5a8247d4a"
format_stamp: "Formatted at 2026-09-25 10:14:49 on dist-test-slave-dzwq"
server_key: "007f6dd3a1bb17f281c63e2d772b30a4"
server_key_iv: "9058e1589782d53c4936c9484f8b0f2f"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:49.068822 15338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "bb18727057dc485db0938ae5a8247d4a"
format_stamp: "Formatted at 2026-09-25 10:14:49 on dist-test-slave-dzwq"
server_key: "007f6dd3a1bb17f281c63e2d772b30a4"
server_key_iv: "9058e1589782d53c4936c9484f8b0f2f"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:49.072384 15338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 10:14:49.088129 15353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:49.088335 15338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 10:14:49.088434 15338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "bb18727057dc485db0938ae5a8247d4a"
format_stamp: "Formatted at 2026-09-25 10:14:49 on dist-test-slave-dzwq"
server_key: "007f6dd3a1bb17f281c63e2d772b30a4"
server_key_iv: "9058e1589782d53c4936c9484f8b0f2f"
server_key_version: "kuduclusterkey@0"
I20260925 10:14:49.088543 15338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:14:49.118423 15338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:14:49.121042 15338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:14:49.121210 15338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:14:49.121498 15338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:14:49.121870 15338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:14:49.121920 15338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:49.121958 15338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:14:49.121975 15338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:14:49.128285 15338 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:35295
I20260925 10:14:49.128719 15338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:14:49.132171 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 15338
I20260925 10:14:49.132277 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:14:49.132486 3683 external_mini_cluster.cc:1428] Setting key 2a5547f98b913dd8abec14075d011a8e
I20260925 10:14:49.132030 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:35295 every 8 connection(s)
Sep 25 10:14:49 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:14:49.144097 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.132929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:55247 (local address 127.3.152.254:37277)
0925 10:14:49.132973 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.132975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.133235 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:14:49.134019 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.134020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.134032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.134063 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:49.134902 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.135685 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.137213 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.137545 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.137593 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.140473 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:49.140474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:49.140475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:49.140482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:49.142405 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:49.142940 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:49.142941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:49.142942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:49.143003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:49.143304 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:49.143305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:49.143306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:49.143449 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:49.143492 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.143871 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.143998 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:14:49.144668 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.132645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37277 (local address 127.3.152.195:55247)
0925 10:14:49.133103 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.133156 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.133779 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.134267 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.134272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.134520 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:14:49.134796 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.134801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.135799 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.135800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.137084 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.137087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.137694 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.137695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.137842 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.138487 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:14:49.138497 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:14:49.140357 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:14:49.142503 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:49.142511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:49.142514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:49.142846 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:49.143074 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:14:49.143074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:14:49.143075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:14:49.143225 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:14:49.143537 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:14:49.143543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:14:49.143712 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.144244 (+ 532us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.144422 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":85,"threads_started":1}
I20260925 10:14:49.145087 15467 heartbeater.cc:347] Connected to a master server at 127.3.152.254:37277
I20260925 10:14:49.145201 15467 heartbeater.cc:464] Registering TS with master...
I20260925 10:14:49.145404 15467 heartbeater.cc:511] Master 127.3.152.254:37277 requested a full tablet report, sending...
I20260925 10:14:49.145756 14983 ts_manager.cc:194] Registered new tserver with Master: bb18727057dc485db0938ae5a8247d4a (127.3.152.195:35295)
I20260925 10:14:49.146273 14983 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:55247
I20260925 10:14:49.155504 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:14:49.165318 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.157388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.3.152.254:37277)
0925 10:14:49.157434 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.157436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.157704 (+ 268us) server_negotiation.cc:407] Connection header received
0925 10:14:49.157802 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.157803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.157814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.157844 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:14:49.158654 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.159423 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.160560 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.160867 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.160918 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.162515 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:14:49.162516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:14:49.162517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:14:49.162524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:14:49.164028 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:49.164410 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:49.164411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:49.164411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:49.164475 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:14:49.164689 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:14:49.164690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:14:49.164690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:14:49.164823 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:14:49.164903 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.165125 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.165206 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:14:49.167042 14983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:14:49.168985 14983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:14:49.180923 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.173467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:42959 (local address 127.0.0.1:35482)
0925 10:14:49.174180 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.174190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.174582 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.177005 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.177006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.177007 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:49.177040 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.177042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.178308 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.178309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.179683 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.179685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.180640 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.180642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.180721 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.180736 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.180784 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.180834 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":68,"threads_started":1}
I20260925 10:14:49.181421 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.173585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.3.152.194:42959)
0925 10:14:49.176788 (+ 3203us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.176792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.176802 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:14:49.176844 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.176845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.176860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.176930 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:49.177106 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.178209 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.179799 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.180548 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.181159 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.181248 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.181277 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.181318 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3156,"thread_start_us":102,"threads_started":1}
I20260925 10:14:49.182129 15267 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 (DEFAULT_TABLE table=test-table [id=20a3739ec7ec4dbbb3ad9025162743ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:49.182499 15267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:49.186211 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.174032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.3.152.193:42083)
0925 10:14:49.174523 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.174530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.174762 (+ 232us) server_negotiation.cc:407] Connection header received
0925 10:14:49.174800 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.174802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.174817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.174884 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:49.175054 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.176141 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.179575 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.180380 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.180504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.180577 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.186071 (+ 5494us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.186138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":50,"threads_started":1}
I20260925 10:14:49.186659 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.173845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:42083 (local address 127.0.0.1:32818)
0925 10:14:49.174662 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.174671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.174712 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.174946 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.174947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.174948 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:49.174987 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.174988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.176224 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.176224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.179309 (+ 3085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.179312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.181275 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.181276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.181333 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.181338 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.181389 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.181431 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":1866,"thread_start_us":63,"threads_started":1}
I20260925 10:14:49.187526 15129 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 (DEFAULT_TABLE table=test-table [id=20a3739ec7ec4dbbb3ad9025162743ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:49.187903 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.173191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:35295 (local address 127.0.0.1:40766)
0925 10:14:49.173352 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.173397 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.173460 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.174240 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.174243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.174246 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:49.174336 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.174338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.176210 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.176211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.178202 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.178205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.187654 (+ 9449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.187656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.187736 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.187741 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.187775 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.187826 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":50,"thread_start_us":102,"threads_started":1}
I20260925 10:14:49.187903 15129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:49.188310 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.173596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.3.152.195:35295)
0925 10:14:49.173761 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.173767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.173778 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:14:49.173825 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.173828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.173845 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.174422 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:49.174501 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.176108 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.180647 (+ 4539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.187494 (+ 6847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.188026 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.188150 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.188180 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.188229 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260925 10:14:49.188519 15485 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Bootstrap starting.
I20260925 10:14:49.188947 15401 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 (DEFAULT_TABLE table=test-table [id=20a3739ec7ec4dbbb3ad9025162743ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:14:49.189311 15401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:14:49.189646 15485 tablet_bootstrap.cc:654] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:49.189965 15485 log.cc:824] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:49.190750 15485 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: No bootstrap required, opened a new log
I20260925 10:14:49.190816 15485 ts_tablet_manager.cc:1397] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:14:49.192955 15485 raft_consensus.cc:348] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.193122 15485 raft_consensus.cc:374] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:49.193157 15485 raft_consensus.cc:729] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667b5b13d5c3433aa26ccf0d9481b34c, State: Initialized, Role: FOLLOWER
I20260925 10:14:49.193272 15485 consensus_queue.cc:260] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.193550 15485 ts_tablet_manager.cc:1428] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:14:49.195557 15488 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: Bootstrap starting.
I20260925 10:14:49.195730 15333 heartbeater.cc:503] Master 127.3.152.254:37277 was elected leader, sending a full tablet report...
I20260925 10:14:49.196314 15488 tablet_bootstrap.cc:654] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:49.196627 15488 log.cc:824] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:49.198028 15490 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Bootstrap starting.
I20260925 10:14:49.199047 15490 tablet_bootstrap.cc:654] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Neither blocks nor log segments found. Creating new log.
I20260925 10:14:49.199368 15490 log.cc:824] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Log is configured to *not* fsync() on all Append() calls
I20260925 10:14:49.199561 15488 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: No bootstrap required, opened a new log
I20260925 10:14:49.199640 15488 ts_tablet_manager.cc:1397] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:14:49.200644 15490 tablet_bootstrap.cc:492] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: No bootstrap required, opened a new log
I20260925 10:14:49.200713 15490 ts_tablet_manager.cc:1397] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:14:49.202319 15488 raft_consensus.cc:348] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.202488 15488 raft_consensus.cc:374] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:49.202520 15488 raft_consensus.cc:729] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c4298a914a45adbfa884b71cb7e997, State: Initialized, Role: FOLLOWER
I20260925 10:14:49.202661 15488 consensus_queue.cc:260] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.202718 15490 raft_consensus.cc:348] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.202868 15490 raft_consensus.cc:374] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:14:49.202903 15490 raft_consensus.cc:729] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb18727057dc485db0938ae5a8247d4a, State: Initialized, Role: FOLLOWER
I20260925 10:14:49.202952 15488 ts_tablet_manager.cc:1428] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 10:14:49.203009 15490 consensus_queue.cc:260] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.203063 15195 heartbeater.cc:503] Master 127.3.152.254:37277 was elected leader, sending a full tablet report...
I20260925 10:14:49.203261 15490 ts_tablet_manager.cc:1428] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:14:49.203367 15467 heartbeater.cc:503] Master 127.3.152.254:37277 was elected leader, sending a full tablet report...
Sep 25 10:14:49 dist-test-slave-dzwq krb5kdc[14507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331288, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:14:49.319202 15492 raft_consensus.cc:482] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:14:49.319388 15492 raft_consensus.cc:504] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.319772 15492 leader_election.cc:288] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb18727057dc485db0938ae5a8247d4a (127.3.152.195:35295), 667b5b13d5c3433aa26ccf0d9481b34c (127.3.152.194:42959)
I20260925 10:14:49.325083 15481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.320163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:33041 (local address 127.3.152.195:35295)
0925 10:14:49.320338 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.320340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.321013 (+ 673us) server_negotiation.cc:407] Connection header received
0925 10:14:49.321218 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.321220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.321231 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.321287 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:49.321595 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.322396 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.324084 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.324647 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.324702 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.324713 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.324945 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.324994 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260925 10:14:49.325364 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.319912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:35295 (local address 127.3.152.193:33041)
0925 10:14:49.320946 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.320957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.321077 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.321343 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.321344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.321346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:49.321439 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.321441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.322651 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.322652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.323952 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.323985 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.324777 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.324778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.324857 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.324861 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.325240 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.325286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"thread_start_us":126,"threads_started":1}
I20260925 10:14:49.325804 15421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6c03eb798a4e72ba1c15c1ddf2f612" candidate_uuid: "c4c4298a914a45adbfa884b71cb7e997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb18727057dc485db0938ae5a8247d4a" is_pre_election: true
I20260925 10:14:49.325982 15421 raft_consensus.cc:2454] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c4298a914a45adbfa884b71cb7e997 in term 0.
I20260925 10:14:49.326432 15080 leader_election.cc:302] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb18727057dc485db0938ae5a8247d4a, c4c4298a914a45adbfa884b71cb7e997; no voters:
I20260925 10:14:49.326577 15492 raft_consensus.cc:2781] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:14:49.326627 15492 raft_consensus.cc:482] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:14:49.326653 15492 raft_consensus.cc:3037] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:49.327610 15492 raft_consensus.cc:504] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.327764 15492 leader_election.cc:288] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [CANDIDATE]: Term 1 election: Requested vote from peers bb18727057dc485db0938ae5a8247d4a (127.3.152.195:35295), 667b5b13d5c3433aa26ccf0d9481b34c (127.3.152.194:42959)
I20260925 10:14:49.328151 15421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6c03eb798a4e72ba1c15c1ddf2f612" candidate_uuid: "c4c4298a914a45adbfa884b71cb7e997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb18727057dc485db0938ae5a8247d4a"
I20260925 10:14:49.328233 15421 raft_consensus.cc:3037] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:49.329134 15421 raft_consensus.cc:2454] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c4298a914a45adbfa884b71cb7e997 in term 1.
I20260925 10:14:49.329382 15080 leader_election.cc:302] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb18727057dc485db0938ae5a8247d4a, c4c4298a914a45adbfa884b71cb7e997; no voters:
I20260925 10:14:49.329499 15492 raft_consensus.cc:2781] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:14:49.329656 15492 raft_consensus.cc:686] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [term 1 LEADER]: Becoming Leader. State: Replica: c4c4298a914a45adbfa884b71cb7e997, State: Running, Role: LEADER
I20260925 10:14:49.329751 15492 consensus_queue.cc:237] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } }
I20260925 10:14:49.330708 14982 catalog_manager.cc:5495] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4c4298a914a45adbfa884b71cb7e997 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "c4c4298a914a45adbfa884b71cb7e997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4c4298a914a45adbfa884b71cb7e997" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 42083 } health_report { overall_health: HEALTHY } } }
I20260925 10:14:49.332187 15482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.320328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:52449 (local address 127.3.152.194:42959)
0925 10:14:49.320369 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.320370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.321312 (+ 942us) server_negotiation.cc:407] Connection header received
0925 10:14:49.321413 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.321414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.321428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.321458 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:14:49.321682 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.322392 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.331072 (+ 8680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.331611 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.331680 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.331690 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.332073 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.332121 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:14:49.332412 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.320004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:42959 (local address 127.3.152.193:52449)
0925 10:14:49.321256 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.321265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:14:49.321339 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:14:49.321522 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:14:49.321523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:14:49.321524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:14:49.321587 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.321588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.322532 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.322533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.330905 (+ 8372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:14:49.330952 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.331885 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.331885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:14:49.331980 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.331985 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:14:49.332270 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 10:14:49.332325 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1235,"thread_start_us":82,"threads_started":1}
I20260925 10:14:49.332862 15287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6c03eb798a4e72ba1c15c1ddf2f612" candidate_uuid: "c4c4298a914a45adbfa884b71cb7e997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" is_pre_election: true
I20260925 10:14:49.333029 15287 raft_consensus.cc:2454] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4c4298a914a45adbfa884b71cb7e997 in term 0.
I20260925 10:14:49.333185 15286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6c03eb798a4e72ba1c15c1ddf2f612" candidate_uuid: "c4c4298a914a45adbfa884b71cb7e997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667b5b13d5c3433aa26ccf0d9481b34c"
I20260925 10:14:49.333269 15286 raft_consensus.cc:3037] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:14:49.334163 15286 raft_consensus.cc:2454] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4c4298a914a45adbfa884b71cb7e997 in term 1.
I20260925 10:14:49.362967 15484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.359439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.3.152.193:42083)
0925 10:14:49.359482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:14:49.359484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:14:49.359491 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:14:49.359531 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:14:49.359532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:14:49.359551 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:14:49.359613 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:14:49.359794 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.360576 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.361766 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:14:49.362056 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:14:49.362132 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:14:49.362354 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:14:49.362499 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:14:49.362587 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:14:49.362841 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:14:49.362866 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:14:49.365442 15421 raft_consensus.cc:1261] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 1 FOLLOWER]: Refusing update from remote peer c4c4298a914a45adbfa884b71cb7e997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:14:49.365442 15286 raft_consensus.cc:1261] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 1 FOLLOWER]: Refusing update from remote peer c4c4298a914a45adbfa884b71cb7e997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:14:49.365804 15492 consensus_queue.cc:1035] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "667b5b13d5c3433aa26ccf0d9481b34c" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:14:49.365994 15492 consensus_queue.cc:1035] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P c4c4298a914a45adbfa884b71cb7e997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb18727057dc485db0938ae5a8247d4a" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:14:49.376070 15502 mvcc.cc:205] Tried to move back new op lower bound from 7333196961237344256 to 7333196961095319552. Current Snapshot: MvccSnapshot[applied={T|T < 7333196961237344256}]
I20260925 10:14:49.380085 15499 mvcc.cc:205] Tried to move back new op lower bound from 7333196961237344256 to 7333196961095319552. Current Snapshot: MvccSnapshot[applied={T|T < 7333196961237344256}]
I20260925 10:14:49.383054 15500 mvcc.cc:205] Tried to move back new op lower bound from 7333196961237344256 to 7333196961095319552. Current Snapshot: MvccSnapshot[applied={T|T < 7333196961237344256}]
I20260925 10:14:49.384493 14982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:14:49.384601 14982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:14:49.385818 14982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b160ad6a39f4bc38a4b2a6609516914: Sending DeleteTablet for 3 replicas of tablet 3f6c03eb798a4e72ba1c15c1ddf2f612
I20260925 10:14:49.386205 15267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:35482
I20260925 10:14:49.386576 15129 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:32818
I20260925 10:14:49.386739 15401 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:14:49 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:40766
I20260925 10:14:49.387414 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 15064
I20260925 10:14:49.390200 15510 tablet_replica.cc:330] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: stopping tablet replica
I20260925 10:14:49.390383 15510 raft_consensus.cc:2229] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:14:49.390594 15510 raft_consensus.cc:2258] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:14:49.390767 15508 tablet_replica.cc:330] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: stopping tablet replica
I20260925 10:14:49.391121 15508 raft_consensus.cc:2229] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:14:49.391475 15510 ts_tablet_manager.cc:1905] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:14:49.391685 15508 raft_consensus.cc:2258] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:14:49.392767 15508 ts_tablet_manager.cc:1905] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:14:49.396260 15510 ts_tablet_manager.cc:1918] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:14:49.396427 15510 log.cc:1196] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/3f6c03eb798a4e72ba1c15c1ddf2f612
I20260925 10:14:49.396822 15510 ts_tablet_manager.cc:1939] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P bb18727057dc485db0938ae5a8247d4a: Deleting consensus metadata
I20260925 10:14:49.397015 15508 ts_tablet_manager.cc:1918] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:14:49.397150 15508 log.cc:1196] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/3f6c03eb798a4e72ba1c15c1ddf2f612
I20260925 10:14:49.397508 15508 ts_tablet_manager.cc:1939] T 3f6c03eb798a4e72ba1c15c1ddf2f612 P 667b5b13d5c3433aa26ccf0d9481b34c: Deleting consensus metadata
I20260925 10:14:49.398084 14967 catalog_manager.cc:4841] TS bb18727057dc485db0938ae5a8247d4a (127.3.152.195:35295): tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 (table test-table [id=20a3739ec7ec4dbbb3ad9025162743ee]) successfully deleted
I20260925 10:14:49.398671 14969 catalog_manager.cc:4841] TS 667b5b13d5c3433aa26ccf0d9481b34c (127.3.152.194:42959): tablet 3f6c03eb798a4e72ba1c15c1ddf2f612 (table test-table [id=20a3739ec7ec4dbbb3ad9025162743ee]) successfully deleted
W20260925 10:14:49.401543 14970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:42083 (error 108)
I20260925 10:14:49.402313 15479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:14:49.402140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:42083 (local address 127.0.0.1:32868)
0925 10:14:49.402209 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 10:14:49.402273 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:14:49.402577 14970 catalog_manager.cc:4584] TS c4c4298a914a45adbfa884b71cb7e997 (127.3.152.193:42083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f6c03eb798a4e72ba1c15c1ddf2f612: Network error: Client connection negotiation failed: client connection to 127.3.152.193:42083: connect: Connection refused (error 111)
I20260925 10:14:49.404114 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 15201
I20260925 10:14:49.409912 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 15338
I20260925 10:14:49.418447 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 14951
I20260925 10:14:49.424968 3683 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:14:49.760353 3683 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:14:49.760453 3683 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:14:50.105360 3683 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:14:50.105 UTC [14537] LOG: received smart shutdown request
2026-09-25 10:14:50.109 UTC [14537] LOG: background worker "logical replication launcher" (PID 14546) exited with exit code 1
2026-09-25 10:14:50.109 UTC [14541] LOG: shutting down
2026-09-25 10:14:50.287 UTC [14537] LOG: database system is shut down
2026-09-25T10:14:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47782 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:14:50 dist-test-slave-dzwq krb5kdc[15520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:14:50 dist-test-slave-dzwq krb5kdc[15520](info): set up 2 sockets
Sep 25 10:14:50 dist-test-slave-dzwq krb5kdc[15520](info): commencing operation
krb5kdc: starting...
W20260925 10:14:51.432662 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:14:51 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:14:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:14:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangeradmin_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangeradmin_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangerlookup_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangerlookup_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
I20260925 10:14:51.604108 3683 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcY4CcD/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/postgres -l logfile start
2026-09-25 10:14:54.207 UTC [15550] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 10:14:54.207 UTC [15550] LOG: listening on IPv4 address "127.3.152.212", port 54965
2026-09-25 10:14:54.211 UTC [15550] LOG: listening on Unix socket "/tmp/.s.PGSQL.54965"
2026-09-25 10:14:54.246 UTC [15553] LOG: database system was shut down at 2026-09-25 10:14:52 UTC
2026-09-25 10:14:54.251 UTC [15550] LOG: database system is ready to accept connections
I20260925 10:14:55.280606 3683 mini_postgres.cc:96] Postgres bound to 54965
2026-09-25 10:14:55.287 UTC [15561] FATAL: database "slave" does not exist
127.3.152.212:54965 - accepting connections
I20260925 10:14:55.287596 3683 mini_ranger.cc:161] Starting Ranger...
I20260925 10:14:55.303074 3683 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 10:14:55.837060 3683 mini_ranger.cc:87] Created ranger Postgres database
I20260925 10:14:55.837150 3683 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin
2026-09-25 10:14:56,118 [I] DB FLAVOR :POSTGRES
2026-09-25 10:14:56,118 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:14:56,118 [I] Checking connection..
2026-09-25 10:14:56,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 10:14:56,379 [I] Checking connection passed.
2026-09-25 10:14:56,380 [I] --------- Verifying version history table ---------
2026-09-25 10:14:56,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:56,625 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:14:56,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:56,890 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 10:14:56,890 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 10:14:56,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 10:14:57.162 UTC [15625] WARNING: there is no transaction in progress
2026-09-25 10:14:57,169 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 10:14:57,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 10:14:57,441 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 10:14:57,442 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:14:57,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 10:14:57,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 10:14:57,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-dzwq.c.gcp-upstream.internal','N') ;"
2026-09-25 10:14:58,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:14:58,432 [I] Table x_portal_user does not exist in database ranger
2026-09-25 10:14:58,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:14:58,726 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 10:14:58,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:14:59,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:14:59,240 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 10:14:59,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 10:15:00.242 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.261 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.279 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.296 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.315 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.334 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.395 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.404 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.422 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.433 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.445 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.460 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.471 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.479 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.487 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:00.494 UTC [15733] WARNING: there is no transaction in progress
2026-09-25 10:15:01,216 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 10:15:01,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 10:15:01,475 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 10:15:01,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 10:15:01,740 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 10:15:01,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 10:15:02,010 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 10:15:02,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 10:15:02,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 10:15:02,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-dzwq.c.gcp-upstream.internal';"
2026-09-25 10:15:02,743 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 10:15:02,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 10:15:02,979 [I] DB_PATCHES have already been applied
I20260925 10:15:02.985813 3683 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcY4CcD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 10:15:02.985888 3683 mini_ranger.cc:193] Using host: 127.3.152.212
I20260925 10:15:02.989120 3683 mini_ranger.cc:227] Ranger admin URL: http://127.3.152.212:59491
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 10:15:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 10:15:04.095412 3683 mini_ranger.cc:160] Time spent starting Ranger: real 8.808s user 0.001s sys 0.008s
Sep 25, 2026 10:15:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 10:15:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 10:15:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 10:15:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 10:15:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 10:15:25.417294 3683 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.152.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.152.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangerkms_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.152.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/rangerkms_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.152.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.152.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.152.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/HTTP_127.3.152.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 10:15:25 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 10:15:25.505604 3683 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 10:15:25.515638 3683 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 10:15:26.455389 3683 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 10:15:26.455497 3683 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms
I20260925 10:15:26.455521 3683 mini_ranger_kms.cc:227] Using postgres at 127.3.152.212:54965
2026-09-25 10:15:26,619 [I] DB FLAVOR :POSTGRES
2026-09-25 10:15:26,620 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 10:15:26,620 [I] Checking connection
2026-09-25 10:15:26,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 10:15:26,867 [I] connection success
2026-09-25 10:15:26,868 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 10:15:26,868 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 10:15:26,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 10:15:27,101 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 10:15:27,101 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 10:15:27,101 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 10:15:27,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.152.212:54965/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 10:15:27,392 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 10:15:27.853094 3683 mini_ranger_kms.cc:310] Created kms service
I20260925 10:15:28.069234 3683 mini_ranger_kms.cc:326] Created kudu user
I20260925 10:15:28.114817 3683 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 10:15:28 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331325, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:28.515867 3683 mini_ranger_kms.cc:384] Added ranger policy
I20260925 10:15:28.516113 3683 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcY4CcD/build/release/bin/postgresql.jar:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcY4CcD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 10:15:28.516144 3683 mini_ranger_kms.cc:240] Using host: 127.3.152.212
I20260925 10:15:28.520334 3683 mini_ranger_kms.cc:276] Ranger KMS PID: 15913
I20260925 10:15:28.520426 3683 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.152.212:43529
Sep 25, 2026 10:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 10:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 10:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 10:15:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 10:15:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 10:15:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 10:15:30.956785 3683 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.451s user 0.000s sys 0.147s
I20260925 10:15:30.956883 3683 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 10:15:30.956928 3683 mini_ranger_kms.cc:401] 127.3.152.212:43529/kms/v1/keys
Sep 25 10:15:31 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 10:15:31 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
Sep 25 10:15:31 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331331, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.152.212@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
Sep 25 10:15:34 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:15:34.215312 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33113
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33113
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:34.299039 15964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:34.299269 15964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:34.299293 15964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:34.300671 15964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:34.300717 15964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:34.300732 15964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:34.300745 15964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:34.300758 15964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:34.302371 15964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.152.212:43529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33113
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:34.302628 15964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:34.302886 15964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:34.308038 15969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:34.308175 15972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:34.308256 15964 server_base.cc:1034] running on GCE node
W20260925 10:15:34.308046 15970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:34.308771 15964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:34.309106 15964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:34.310324 15964 hybrid_clock.cc:648] HybridClock initialized: now 1790331334310294 us; error 45 us; skew 500 ppm
Sep 25 10:15:34 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:34.318035 15964 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:34.318586 15964 webserver.cc:460] Webserver started at http://127.3.152.254:36009/ using document root <none> and password file <none>
I20260925 10:15:34.318841 15964 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:34.318882 15964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:34.319036 15964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:15:34 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:34.383265 15964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "8c4a57a4d28040a6ba137d356998ca58"
format_stamp: "Formatted at 2026-09-25 10:15:34 on dist-test-slave-dzwq"
server_key: "b859962b64786faf32479dfcab9c2f3c"
server_key_iv: "5e5e6fce38568d1153f990b9e1dd1dba"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:34.383745 15964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "8c4a57a4d28040a6ba137d356998ca58"
format_stamp: "Formatted at 2026-09-25 10:15:34 on dist-test-slave-dzwq"
server_key: "b859962b64786faf32479dfcab9c2f3c"
server_key_iv: "5e5e6fce38568d1153f990b9e1dd1dba"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:34.385711 15964 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 10:15:34.400513 15979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:34.400775 15964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:34.400868 15964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8c4a57a4d28040a6ba137d356998ca58"
format_stamp: "Formatted at 2026-09-25 10:15:34 on dist-test-slave-dzwq"
server_key: "b859962b64786faf32479dfcab9c2f3c"
server_key_iv: "5e5e6fce38568d1153f990b9e1dd1dba"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:34.400956 15964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:34.415350 15964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:34.417621 15964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:34.417788 15964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:34.423185 15964 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:33113
I20260925 10:15:34.423237 16031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:33113 every 8 connection(s)
I20260925 10:15:34.423686 15964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:34.424458 16032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:34.426831 16032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Bootstrap starting.
I20260925 10:15:34.427430 16032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:34.427738 16032 log.cc:824] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:34.428471 16032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: No bootstrap required, opened a new log
I20260925 10:15:34.430505 16032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:34.430740 16032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:34.430791 16032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4a57a4d28040a6ba137d356998ca58, State: Initialized, Role: FOLLOWER
I20260925 10:15:34.430950 16032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:34.431034 16032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:34.431077 16032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:34.431124 16032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:34.431763 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 15964
I20260925 10:15:34.431767 16032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:34.431855 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:15:34.431875 16032 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4a57a4d28040a6ba137d356998ca58; no voters:
I20260925 10:15:34.432077 16032 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:15:34.432106 3683 external_mini_cluster.cc:1428] Setting key 9273bc014e524585186db7d681b60516
I20260925 10:15:34.432139 16035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:34.432863 16035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 LEADER]: Becoming Leader. State: Replica: 8c4a57a4d28040a6ba137d356998ca58, State: Running, Role: LEADER
I20260925 10:15:34.433038 16035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:34.433084 16032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:34.434998 16039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4a57a4d28040a6ba137d356998ca58. Latest consensus state: current_term: 1 leader_uuid: "8c4a57a4d28040a6ba137d356998ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } } }
I20260925 10:15:34.435097 16039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:34.435467 16037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c4a57a4d28040a6ba137d356998ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } } }
I20260925 10:15:34.435514 16052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:34.435549 16037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:34.435938 16032 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:15:34.436188 16052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:15:34 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:34.444165 16052 catalog_manager.cc:1347] Generated new cluster ID: d202df2a7321461285c8aaf5e71f9712
I20260925 10:15:34.444278 16052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:15:34.449373 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:34.432598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.3.152.254:33113)
0925 10:15:34.433180 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:34.433191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:34.433223 (+ 32us) server_negotiation.cc:407] Connection header received
0925 10:15:34.433347 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:34.433352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:34.433395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:34.433563 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:34.434315 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:34.435728 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:34.436914 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:34.437523 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:34.437643 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:34.441964 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:34.442023 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:34.442023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:34.442154 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:34.446870 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:34.447362 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:34.447366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:34.447366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:34.447509 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:34.447755 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:34.447755 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:34.447756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:34.448149 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:34.448843 (+ 694us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:34.448924 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 10:15:34.449089 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":268,"threads_started":1}
I20260925 10:15:34.455770 16052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:15:34.456630 16052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:34.462967 16052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Generated new TSK 0
I20260925 10:15:34.463166 16052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:15:35.180301 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:35.277738 16078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:35.278050 16078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:35.278102 16078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:35.279330 16078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:35.279376 16078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:35.279402 16078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:15:35.280937 16078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:35.281172 16078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:35.281395 16078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:35.287539 16084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.287511 16083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.287712 16086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:35.287753 16078 server_base.cc:1034] running on GCE node
I20260925 10:15:35.288103 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:35.288367 16078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:35.289585 16078 hybrid_clock.cc:648] HybridClock initialized: now 1790331335289558 us; error 39 us; skew 500 ppm
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:35.293057 16078 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:15:35.293584 16078 webserver.cc:460] Webserver started at http://127.3.152.193:46457/ using document root <none> and password file <none>
I20260925 10:15:35.293845 16078 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:35.293919 16078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:35.294130 16078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:35.309650 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "576bfcf156164a67aae5c49b0e21c1ec"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "806cf8fb9ae844815a54e323dffa5305"
server_key_iv: "4061e346f23dd8e02181e3e5689de778"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.310112 16078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "576bfcf156164a67aae5c49b0e21c1ec"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "806cf8fb9ae844815a54e323dffa5305"
server_key_iv: "4061e346f23dd8e02181e3e5689de778"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.312119 16078 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 10:15:35.325605 16093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.325872 16078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:35.325994 16078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "576bfcf156164a67aae5c49b0e21c1ec"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "806cf8fb9ae844815a54e323dffa5305"
server_key_iv: "4061e346f23dd8e02181e3e5689de778"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.326122 16078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:35.337699 16078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:35.340479 16078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:35.340772 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:35.341100 16078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:35.341603 16078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:35.341667 16078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.341711 16078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:35.341768 16078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.349910 16078 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35147
I20260925 10:15:35.350436 16078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:15:35.351076 16208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35147 every 8 connection(s)
I20260925 10:15:35.357694 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16078
I20260925 10:15:35.357813 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:15:35.358045 3683 external_mini_cluster.cc:1428] Setting key aa46d2d1b0c26eab707ec909f5d0792f
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:35.363186 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.351806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.193:39553)
0925 10:15:35.352104 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:35.352153 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:35.352792 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:35.353140 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:35.353145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:35.353394 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:35.353669 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.353676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.354734 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.354735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.356048 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.356051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.356620 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.356621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.356760 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.357551 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:35.357561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:35.359306 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:35.361348 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.361355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.361358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.361662 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.361957 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.361959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.361959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.362117 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.362435 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:35.362441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:35.362631 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 10:15:35.362740 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 10:15:35.362912 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":87,"threads_started":1}
I20260925 10:15:35.364085 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.351842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:39553 (local address 127.3.152.254:33113)
0925 10:15:35.352019 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:35.352022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:35.352272 (+ 250us) server_negotiation.cc:407] Connection header received
0925 10:15:35.352934 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:35.352936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:35.352952 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:35.353047 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:35.353789 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.354614 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.356156 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.356468 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.356512 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.359399 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:35.359400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:35.359401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:35.359408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:35.361251 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.361761 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.361763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.361763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.361845 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.362208 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.362209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.362210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.362353 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:35.363798 (+ 1445us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:35.363847 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 10:15:35.363944 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260925 10:15:35.364737 16209 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:35.364841 16209 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:35.365031 16209 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:35.365605 15996 ts_manager.cc:194] Registered new tserver with Master: 576bfcf156164a67aae5c49b0e21c1ec (127.3.152.193:35147)
I20260925 10:15:35.366524 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:39553
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:15:35.389076 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:35.496313 16215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:35.496557 16215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:35.496599 16215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:35.497852 16215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:35.497936 16215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:35.497987 16215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:15:35.499607 16215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:35.499928 16215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:35.500227 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:35.505415 16225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.505424 16222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.505824 16223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:35.506448 16215 server_base.cc:1034] running on GCE node
I20260925 10:15:35.506659 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:35.506944 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:35.508092 16215 hybrid_clock.cc:648] HybridClock initialized: now 1790331335508069 us; error 40 us; skew 500 ppm
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:35.510674 16215 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:15:35.511147 16215 webserver.cc:460] Webserver started at http://127.3.152.194:35043/ using document root <none> and password file <none>
I20260925 10:15:35.511406 16215 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:35.511477 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:35.511620 16215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:35.532907 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "b794c9d9d0d347a193f154f8dac2bbf9"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "7786fe60d97036898d29a57f51c2ee32"
server_key_iv: "f686157f81a83d831acd9fcf90e87c8e"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.533521 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "b794c9d9d0d347a193f154f8dac2bbf9"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "7786fe60d97036898d29a57f51c2ee32"
server_key_iv: "f686157f81a83d831acd9fcf90e87c8e"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.539258 16215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:35.561466 16232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.563442 16215 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260925 10:15:35.563581 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "b794c9d9d0d347a193f154f8dac2bbf9"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "7786fe60d97036898d29a57f51c2ee32"
server_key_iv: "f686157f81a83d831acd9fcf90e87c8e"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.563694 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:35.576023 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:35.578667 16215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:35.578908 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:35.579231 16215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:35.579703 16215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:35.579799 16215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.579878 16215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:35.580022 16215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.587119 16215 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:40901
I20260925 10:15:35.587689 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:15:35.589206 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16215
I20260925 10:15:35.589329 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:15:35.589587 3683 external_mini_cluster.cc:1428] Setting key 5dacd44af35a1ca3a7038f557be8c418
I20260925 10:15:35.590359 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:40901 every 8 connection(s)
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:15:35.613804 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.591931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:37949 (local address 127.3.152.254:33113)
0925 10:15:35.592013 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:35.592015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:35.596326 (+ 4311us) server_negotiation.cc:407] Connection header received
0925 10:15:35.597115 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:35.597117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:35.597135 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:35.597170 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:35.598132 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.598986 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.600727 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.601051 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.601117 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.609808 (+ 8691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:35.609811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:35.609811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:35.609822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:35.611854 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.612505 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.612507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.612507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.612578 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.612917 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.612918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.612918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.613091 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:35.613148 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:35.613578 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 10:15:35.613692 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:15:35.614704 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.591623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.194:37949)
0925 10:15:35.596153 (+ 4530us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:35.596219 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:35.596956 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:35.597403 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:35.597408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:35.597691 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:35.597987 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.597995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.599147 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.599148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.600615 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.600618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.601238 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.601239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.601393 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.607561 (+ 6168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:35.607576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:35.609647 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:35.612008 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.612018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.612021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.612393 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.612666 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.612667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.612668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.612831 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.613186 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:35.613192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:35.613391 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 10:15:35.614224 (+ 833us) client_negotiation.cc:240] Negotiation successful
0925 10:15:35.614426 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4466,"thread_start_us":99,"threads_started":1}
I20260925 10:15:35.615464 16346 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:35.615576 16346 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:35.615813 16346 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:35.616329 15996 ts_manager.cc:194] Registered new tserver with Master: b794c9d9d0d347a193f154f8dac2bbf9 (127.3.152.194:40901)
I20260925 10:15:35.616871 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:37949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:15:35.643321 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:35.763907 16351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:35.764232 16351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:35.764267 16351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:35.765890 16351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:35.765945 16351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:35.765985 16351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:15:35.768193 16351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:35.768428 16351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:35.768705 16351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:35.776773 16360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.776773 16358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.776916 16357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:35.783317 16351 server_base.cc:1034] running on GCE node
I20260925 10:15:35.783682 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:35.783982 16351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:35.785149 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790331335785126 us; error 32 us; skew 500 ppm
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:35.789839 16351 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:15:35.790333 16351 webserver.cc:460] Webserver started at http://127.3.152.195:35119/ using document root <none> and password file <none>
I20260925 10:15:35.790589 16351 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:35.790639 16351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:35.790760 16351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:35.810024 16351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "bad6d380d13c43ee9773b9afe47aab38"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "3c5666f9194cb74b505411d78af3835b"
server_key_iv: "775f288a659463b597d47e18b4a761ee"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.811750 16351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "bad6d380d13c43ee9773b9afe47aab38"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "3c5666f9194cb74b505411d78af3835b"
server_key_iv: "775f288a659463b597d47e18b4a761ee"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.815846 16351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260925 10:15:35.833817 16367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.834117 16351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 10:15:35.834229 16351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "bad6d380d13c43ee9773b9afe47aab38"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "3c5666f9194cb74b505411d78af3835b"
server_key_iv: "775f288a659463b597d47e18b4a761ee"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:35.834332 16351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:35.847031 16351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:35.849664 16351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:35.849864 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:35.850116 16351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:35.850508 16351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:35.850600 16351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.850669 16351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:35.850724 16351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:35.857708 16351 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:45399
I20260925 10:15:35.858208 16351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:15:35.861199 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:45399 every 8 connection(s)
I20260925 10:15:35.863165 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16351
I20260925 10:15:35.863256 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:15:35.863423 3683 external_mini_cluster.cc:1428] Setting key 167c4cd333669d617a7e3bfda0d9a971
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:35.872843 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.860911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:48695 (local address 127.3.152.254:33113)
0925 10:15:35.860956 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:35.860959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:35.861547 (+ 588us) server_negotiation.cc:407] Connection header received
0925 10:15:35.862238 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:35.862240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:35.862253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:35.862282 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:35.864119 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.864879 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.866399 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.866678 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.866717 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.869351 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:35.869353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:35.869353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:35.869361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:35.871017 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.871598 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.871600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.871600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.871663 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:35.872056 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:35.872057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:35.872057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:35.872199 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:35.872269 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:35.872668 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 10:15:35.872755 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:15:35.873102 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:35.860635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.195:48695)
0925 10:15:35.861422 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:35.861462 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:35.862119 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:35.862463 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:35.862467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:35.862720 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:35.863857 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.863865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.865104 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.865105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.866300 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:35.866303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:35.866818 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:35.866819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:35.866949 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:35.867606 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:35.867616 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:35.869246 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:35.871130 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.871137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.871140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.871487 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.871756 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:35.871758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:35.871758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:35.871921 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:35.872282 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:35.872288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:35.872480 (+ 192us) client_negotiation.cc:769] Sending connection context
0925 10:15:35.872611 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:15:35.872804 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":89,"threads_started":1}
I20260925 10:15:35.873797 16481 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:35.873916 16481 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:35.874136 16481 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:35.874650 15996 ts_manager.cc:194] Registered new tserver with Master: bad6d380d13c43ee9773b9afe47aab38 (127.3.152.195:45399)
I20260925 10:15:35.875166 15996 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:48695
I20260925 10:15:35.876135 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:15:35.877008 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16078
I20260925 10:15:35.883579 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16215
I20260925 10:15:35.891103 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16351
I20260925 10:15:35.897464 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 15964
I20260925 10:15:35.903213 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33113
--webserver_interface=127.3.152.254
--webserver_port=36009
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33113
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:35.986502 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:35.986747 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:35.986784 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:35.988029 16484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:35.988101 16484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:35.988122 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:35.988143 16484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:35.988173 16484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:35.989729 16484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33113
--ranger_config_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=36009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:35.990002 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:35.990233 16484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:35.996095 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.996134 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:35.996296 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:35.996222 16484 server_base.cc:1034] running on GCE node
I20260925 10:15:35.996615 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:35.996901 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:35.998068 16484 hybrid_clock.cc:648] HybridClock initialized: now 1790331335998054 us; error 31 us; skew 500 ppm
Sep 25 10:15:35 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:36.000583 16484 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:36.001094 16484 webserver.cc:460] Webserver started at http://127.3.152.254:36009/ using document root <none> and password file <none>
I20260925 10:15:36.001322 16484 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:36.001374 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:36.056463 16484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:36.057371 16500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.057523 16484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:36.057605 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8c4a57a4d28040a6ba137d356998ca58"
format_stamp: "Formatted at 2026-09-25 10:15:34 on dist-test-slave-dzwq"
server_key: "b859962b64786faf32479dfcab9c2f3c"
server_key_iv: "5e5e6fce38568d1153f990b9e1dd1dba"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:36.058084 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:36.080722 16484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:36.082938 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:36.083098 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:36.087980 16484 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:33113
I20260925 10:15:36.088070 16553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:33113 every 8 connection(s)
I20260925 10:15:36.088467 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:36.089753 16554 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:15:36.090080 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16484
I20260925 10:15:36.090390 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:35147
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:33113
--webserver_port=46457
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:36.090772 16554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Bootstrap starting.
I20260925 10:15:36.099211 16554 log.cc:824] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:36.101193 16554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:15:36.101502 16554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Bootstrap complete.
I20260925 10:15:36.104132 16554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:36.104378 16554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c4a57a4d28040a6ba137d356998ca58, State: Initialized, Role: FOLLOWER
I20260925 10:15:36.104543 16554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:36.104620 16554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:36.104671 16554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:36.104719 16554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:15:36.105772 16554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:36.105861 16554 leader_election.cc:302] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c4a57a4d28040a6ba137d356998ca58; no voters:
I20260925 10:15:36.106058 16554 leader_election.cc:288] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:15:36.106093 16558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:15:36.106258 16558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [term 2 LEADER]: Becoming Leader. State: Replica: 8c4a57a4d28040a6ba137d356998ca58, State: Running, Role: LEADER
I20260925 10:15:36.106336 16554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:36.106413 16558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } }
I20260925 10:15:36.106745 16559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c4a57a4d28040a6ba137d356998ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } } }
I20260925 10:15:36.106824 16559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:36.107110 16560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c4a57a4d28040a6ba137d356998ca58. Latest consensus state: current_term: 2 leader_uuid: "8c4a57a4d28040a6ba137d356998ca58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c4a57a4d28040a6ba137d356998ca58" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33113 } } }
I20260925 10:15:36.107195 16560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:36.107306 16564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:36.108042 16564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:36.108240 16564 catalog_manager.cc:1259] Loaded cluster ID: d202df2a7321461285c8aaf5e71f9712
I20260925 10:15:36.108315 16564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:15:36.108414 16554 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 10:15:36.109549 16564 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:36.109730 16564 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Loaded TSK: 0
I20260925 10:15:36.109956 16564 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:15:36.205785 16556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:36.206044 16556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:36.206084 16556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:36.207795 16556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:36.207921 16556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:36.208010 16556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:15:36.209811 16556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:35147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=46457
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:36.210099 16556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:36.210377 16556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:36.216166 16595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:36.216179 16593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:36.216413 16592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:36.216758 16556 server_base.cc:1034] running on GCE node
I20260925 10:15:36.216979 16556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:36.217299 16556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:36.218494 16556 hybrid_clock.cc:648] HybridClock initialized: now 1790331336218430 us; error 40 us; skew 500 ppm
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:36.224426 16556 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:15:36.225019 16556 webserver.cc:460] Webserver started at http://127.3.152.193:46457/ using document root <none> and password file <none>
I20260925 10:15:36.225313 16556 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:36.225387 16556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:36.251796 16556 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260925 10:15:36.253409 16603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.253636 16556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:36.253747 16556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "576bfcf156164a67aae5c49b0e21c1ec"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "806cf8fb9ae844815a54e323dffa5305"
server_key_iv: "4061e346f23dd8e02181e3e5689de778"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:36.254174 16556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:36.265501 16556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:36.267771 16556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:36.267948 16556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:36.268282 16556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:36.268658 16556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:36.268720 16556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.268764 16556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:36.268791 16556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.276031 16556 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35147
I20260925 10:15:36.276513 16556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:15:36.276641 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16556
I20260925 10:15:36.276949 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:40901
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:33113
--webserver_port=35043
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:36.277163 16716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35147 every 8 connection(s)
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:36.293515 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.278722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.193:42027)
0925 10:15:36.279612 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:36.279666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:36.280643 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:36.281374 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:36.281379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:36.281647 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:36.281956 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.281963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.283826 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.283827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.285195 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.285198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.286217 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.286218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.286352 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.287052 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:36.287063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:36.288909 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:36.291522 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.291530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.291533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.291853 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.292173 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.292174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.292174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.292312 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.292837 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:36.292843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:36.293001 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 10:15:36.293101 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:15:36.293262 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":88,"threads_started":1}
I20260925 10:15:36.294324 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.279132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42027 (local address 127.3.152.254:33113)
0925 10:15:36.280084 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:36.280095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:36.280140 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:15:36.280961 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:36.280969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:36.281037 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:36.281178 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:36.282117 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.283695 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.285309 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.285978 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.286091 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.289047 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:36.289052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:36.289052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:36.289066 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:36.291418 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.291956 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.291976 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.291976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.292093 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.292388 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.292389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.292390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.292719 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:36.292806 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:36.293978 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0925 10:15:36.294116 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":84,"threads_started":1}
I20260925 10:15:36.295192 16717 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:36.295298 16717 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.295536 16717 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.296591 16717 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33113 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:15:36.296674 16717 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.296774 16717 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:36.297113 16717 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.297255 16717 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.297648 16717 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:15:36.381899 16719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:36.382112 16719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:36.382138 16719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:36.383388 16719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:36.383432 16719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:36.383460 16719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:15:36.385130 16719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:40901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=35043
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:36.385390 16719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:36.385630 16719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:36.390847 16729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:36.390856 16726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:36.391109 16727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:36.391359 16719 server_base.cc:1034] running on GCE node
I20260925 10:15:36.391594 16719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:36.391870 16719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:36.393074 16719 hybrid_clock.cc:648] HybridClock initialized: now 1790331336393050 us; error 35 us; skew 500 ppm
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:36.396209 16719 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:15:36.396770 16719 webserver.cc:460] Webserver started at http://127.3.152.194:35043/ using document root <none> and password file <none>
I20260925 10:15:36.397003 16719 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:36.397082 16719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:36.420758 16719 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 10:15:36.421926 16737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.422169 16719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:36.422260 16719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "b794c9d9d0d347a193f154f8dac2bbf9"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "7786fe60d97036898d29a57f51c2ee32"
server_key_iv: "f686157f81a83d831acd9fcf90e87c8e"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:36.422770 16719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:36.444630 16719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:36.446990 16719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:36.447222 16719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:36.447548 16719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:36.448051 16719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:36.448110 16719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.448159 16719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:36.448186 16719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.454941 16719 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:40901
I20260925 10:15:36.455026 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:40901 every 8 connection(s)
I20260925 10:15:36.455401 16719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:15:36.464996 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16719
I20260925 10:15:36.465307 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.152.212:43529/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:45399
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:33113
--webserver_port=35119
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:36.470898 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.456503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:33133 (local address 127.3.152.254:33113)
0925 10:15:36.456556 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:36.456558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:36.460272 (+ 3714us) server_negotiation.cc:407] Connection header received
0925 10:15:36.460871 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:36.460872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:36.460888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:36.460943 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:36.461526 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.462065 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.463182 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.463377 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.463464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.467238 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:36.467240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:36.467240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:36.467248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:36.469129 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.469660 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.469662 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.469662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.469726 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.470029 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.470031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.470031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.470203 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:36.470251 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:36.470698 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 10:15:36.470798 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:15:36.471310 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.456255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.194:33133)
0925 10:15:36.460103 (+ 3848us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:36.460167 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:36.460756 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:36.461019 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:36.461023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:36.461227 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:36.461429 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.461433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.462168 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.462169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.463062 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.463065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.463492 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.463494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.463613 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.464190 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:36.464198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:36.467102 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:36.469234 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.469241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.469243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.469571 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.469798 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.469799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.469799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.469945 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.470303 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:36.470311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:36.470504 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 10:15:36.470637 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:15:36.470828 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3789,"thread_start_us":83,"threads_started":1}
I20260925 10:15:36.472045 16851 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:36.472173 16851 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.472395 16851 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.473296 16851 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33113 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:15:36.473373 16851 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.473448 16851 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:36.473834 16851 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.473917 16851 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.474280 16851 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:15:36.560956 16854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:36.561326 16854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:36.561430 16854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:36.563200 16854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:36.563320 16854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:36.563406 16854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:15:36.565771 16854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.152.212:43529/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:45399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=35119
--tserver_master_addrs=127.3.152.254:33113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:36.566232 16854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:36.566569 16854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:36.573208 16859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:36.573228 16862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:36.573747 16854 server_base.cc:1034] running on GCE node
W20260925 10:15:36.574024 16860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:36.574357 16854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:36.574653 16854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:36.575826 16854 hybrid_clock.cc:648] HybridClock initialized: now 1790331336575419 us; error 421 us; skew 500 ppm
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:36.579188 16854 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:15:36.579761 16854 webserver.cc:460] Webserver started at http://127.3.152.195:35119/ using document root <none> and password file <none>
I20260925 10:15:36.580085 16854 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:36.580181 16854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:36.605732 16854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 10:15:36.606961 16870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.607183 16854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:36.607273 16854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "bad6d380d13c43ee9773b9afe47aab38"
format_stamp: "Formatted at 2026-09-25 10:15:35 on dist-test-slave-dzwq"
server_key: "3c5666f9194cb74b505411d78af3835b"
server_key_iv: "775f288a659463b597d47e18b4a761ee"
server_key_version: "kuduclusterkey@0"
I20260925 10:15:36.607781 16854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:36.624359 16854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:36.627004 16854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:36.627245 16854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:36.627597 16854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:36.628067 16854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:36.628173 16854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.628253 16854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:36.628312 16854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:36.636029 16854 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:45399
I20260925 10:15:36.636332 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:45399 every 8 connection(s)
I20260925 10:15:36.636650 16854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:15:36.642409 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 16854
Sep 25 10:15:36 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:36.647411 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.637483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:33113 (local address 127.3.152.195:33617)
0925 10:15:36.637651 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:36.637694 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:36.638317 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:36.638815 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:36.638820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:36.639087 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:36.639359 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.639367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.640136 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.640137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.641021 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:36.641023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.641425 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.641426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:36.641548 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.642107 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:36.642115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:36.643724 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:36.645705 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.645712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.645715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.646016 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.646272 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:36.646273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:36.646273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:36.646416 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:36.646717 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:36.646724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:36.646894 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 10:15:36.647008 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 10:15:36.647170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 10:15:36.647783 16721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.637499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:33617 (local address 127.3.152.254:33113)
0925 10:15:36.637556 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:36.637558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:36.638609 (+ 1051us) server_negotiation.cc:407] Connection header received
0925 10:15:36.638638 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:36.638640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:36.638650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:36.638732 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:36.639446 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.640046 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.641120 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.641321 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.641397 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.643821 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:36.643823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:36.643824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:36.643832 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:36.645624 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.646109 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.646111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.646111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.646181 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.646491 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.646492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.646493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.646637 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:36.646715 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:36.647631 (+ 916us) server_negotiation.cc:299] Negotiation successful
0925 10:15:36.647712 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:15:36.648334 16985 heartbeater.cc:347] Connected to a master server at 127.3.152.254:33113
I20260925 10:15:36.648432 16985 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.648643 16985 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.649286 16985 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:33113 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 10:15:36.649358 16985 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.649425 16985 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:36.649804 16985 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:36.649881 16985 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
W20260925 10:15:36.650249 16985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:15:36.653359 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:36.642834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.3.152.254:33113)
0925 10:15:36.643019 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:36.643021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:36.644101 (+ 1080us) server_negotiation.cc:407] Connection header received
0925 10:15:36.644123 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:36.644124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:36.644169 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:36.644230 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:36.645728 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.646474 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.648131 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:36.648433 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:36.648496 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:36.650366 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:36.650368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:36.650368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:36.650374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:36.651834 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.652308 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.652309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.652309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.652390 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:36.652658 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:36.652660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:36.652660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:36.652796 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:36.652966 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:36.653054 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:15:36.653151 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
Sep 25 10:15:37 dist-test-slave-dzwq krb5kdc[15520](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790331337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:37.297888 16717 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:37.298079 16717 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:37.298512 16517 ts_manager.cc:194] Registered new tserver with Master: 576bfcf156164a67aae5c49b0e21c1ec (127.3.152.193:35147)
I20260925 10:15:37.299330 16517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:42027
I20260925 10:15:37.474485 16851 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:37.474668 16851 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:37.475190 16517 ts_manager.cc:194] Registered new tserver with Master: b794c9d9d0d347a193f154f8dac2bbf9 (127.3.152.194:40901)
I20260925 10:15:37.475730 16517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:33133
Sep 25 10:15:37 dist-test-slave-dzwq krb5kdc[15520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331337, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for HTTP/127.3.152.212@KRBTEST.COM
I20260925 10:15:37.650440 16985 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:37.650605 16985 heartbeater.cc:511] Master 127.3.152.254:33113 requested a full tablet report, sending...
I20260925 10:15:37.650998 16517 ts_manager.cc:194] Registered new tserver with Master: bad6d380d13c43ee9773b9afe47aab38 (127.3.152.195:45399)
I20260925 10:15:37.651512 16517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:33617
I20260925 10:15:37.654386 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:15:37.664222 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.655336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.3.152.254:33113)
0925 10:15:37.655452 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.655455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.656199 (+ 744us) server_negotiation.cc:407] Connection header received
0925 10:15:37.656258 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.656258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.656272 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.656346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:37.657002 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.657817 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.658861 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.659201 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.659269 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.660893 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:37.660894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:37.660894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:37.660902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:37.662831 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:37.663258 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:37.663260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:37.663260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:37.663323 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:37.663539 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:37.663540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:37.663540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:37.663682 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:37.663782 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.663915 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.664020 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260925 10:15:37.666507 16517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:15:37.668757 16517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:15:37.677042 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.671713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:40901 (local address 127.0.0.1:58740)
0925 10:15:37.672345 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.672385 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:37.672463 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:37.672666 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:37.672669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:37.672671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:37.672745 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.672747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.674239 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.674240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.675542 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.675545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.676697 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.676698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.676805 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.676821 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 10:15:37.676896 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:15:37.676951 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":79,"threads_started":1}
I20260925 10:15:37.677055 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.672124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.3.152.194:40901)
0925 10:15:37.672248 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.672252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.672482 (+ 230us) server_negotiation.cc:407] Connection header received
0925 10:15:37.672534 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.672537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.672554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.672644 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:37.672885 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.673960 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.675646 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.676476 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.676562 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.676629 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.676923 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.676970 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260925 10:15:37.678296 16785 tablet_service.cc:1467] Processing CreateTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 (DEFAULT_TABLE table=test-table [id=135517749f6647c786cfa216c2d5a42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:37.678679 16785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a9a89d678a4df9a0f4c974852becc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:37.681879 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.671994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.3.152.195:45399)
0925 10:15:37.672153 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.672159 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.677627 (+ 5468us) server_negotiation.cc:407] Connection header received
0925 10:15:37.677835 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.677838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.677854 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.677917 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:37.678123 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.679175 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.680710 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.681442 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.681488 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.681536 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.681761 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.681800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260925 10:15:37.682348 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.671719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:45399 (local address 127.0.0.1:49978)
0925 10:15:37.677558 (+ 5839us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.677570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:37.677744 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:37.678022 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:37.678023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:37.678024 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:37.678055 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.678056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.679511 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.679511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.680608 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.680610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.681597 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.681598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.681671 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.681674 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:15:37.682238 (+ 564us) client_negotiation.cc:240] Negotiation successful
0925 10:15:37.682276 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5827,"thread_start_us":88,"threads_started":1}
I20260925 10:15:37.682997 16902 tablet_service.cc:1467] Processing CreateTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 (DEFAULT_TABLE table=test-table [id=135517749f6647c786cfa216c2d5a42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:37.683329 16902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a9a89d678a4df9a0f4c974852becc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:37.683897 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.671941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35147 (local address 127.0.0.1:48282)
0925 10:15:37.680114 (+ 8173us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.680125 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:37.680191 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:37.680441 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:37.680441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:37.680443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:37.680484 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.680485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.681655 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.681655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.682716 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.682719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.683665 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.683666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.683730 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.683734 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:15:37.683795 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:15:37.683831 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8157,"thread_start_us":347,"threads_started":1}
I20260925 10:15:37.684137 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.672047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.3.152.193:35147)
0925 10:15:37.680054 (+ 8007us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.680060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.680245 (+ 185us) server_negotiation.cc:407] Connection header received
0925 10:15:37.680280 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.680282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.680297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.680361 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:37.680546 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.681562 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.682807 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.683547 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.683635 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.683693 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.684025 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.684068 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7966,"thread_start_us":41,"threads_started":1}
I20260925 10:15:37.686708 17019 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Bootstrap starting.
I20260925 10:15:37.687340 17019 tablet_bootstrap.cc:654] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:37.687501 17018 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Bootstrap starting.
I20260925 10:15:37.687811 17019 log.cc:824] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:37.688244 17018 tablet_bootstrap.cc:654] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:37.688593 17018 log.cc:824] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:37.688653 17019 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: No bootstrap required, opened a new log
I20260925 10:15:37.688719 17019 ts_tablet_manager.cc:1397] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:15:37.691023 17019 raft_consensus.cc:348] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.691187 17019 raft_consensus.cc:374] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:37.691218 17019 raft_consensus.cc:729] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad6d380d13c43ee9773b9afe47aab38, State: Initialized, Role: FOLLOWER
I20260925 10:15:37.691339 17019 consensus_queue.cc:260] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.691596 17019 ts_tablet_manager.cc:1428] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:15:37.691675 16985 heartbeater.cc:503] Master 127.3.152.254:33113 was elected leader, sending a full tablet report...
I20260925 10:15:37.692684 16651 tablet_service.cc:1467] Processing CreateTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 (DEFAULT_TABLE table=test-table [id=135517749f6647c786cfa216c2d5a42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:37.693024 16651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22a9a89d678a4df9a0f4c974852becc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:37.693415 17018 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: No bootstrap required, opened a new log
I20260925 10:15:37.693480 17018 ts_tablet_manager.cc:1397] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 10:15:37.696026 17018 raft_consensus.cc:348] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.696187 17018 raft_consensus.cc:374] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:37.696221 17018 raft_consensus.cc:729] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b794c9d9d0d347a193f154f8dac2bbf9, State: Initialized, Role: FOLLOWER
I20260925 10:15:37.696343 17018 consensus_queue.cc:260] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.696663 17018 ts_tablet_manager.cc:1428] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:15:37.696828 16851 heartbeater.cc:503] Master 127.3.152.254:33113 was elected leader, sending a full tablet report...
I20260925 10:15:37.696964 17024 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Bootstrap starting.
I20260925 10:15:37.697640 17024 tablet_bootstrap.cc:654] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:37.698009 17024 log.cc:824] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:37.698885 17024 tablet_bootstrap.cc:492] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: No bootstrap required, opened a new log
I20260925 10:15:37.698961 17024 ts_tablet_manager.cc:1397] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:15:37.701153 17024 raft_consensus.cc:348] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.701332 17024 raft_consensus.cc:374] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:37.701406 17024 raft_consensus.cc:729] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576bfcf156164a67aae5c49b0e21c1ec, State: Initialized, Role: FOLLOWER
I20260925 10:15:37.701539 17024 consensus_queue.cc:260] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.701839 17024 ts_tablet_manager.cc:1428] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:15:37.701853 16717 heartbeater.cc:503] Master 127.3.152.254:33113 was elected leader, sending a full tablet report...
W20260925 10:15:37.706635 16852 tablet.cc:2369] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:15:37.773891 17023 raft_consensus.cc:482] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:37.774062 17023 raft_consensus.cc:504] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.774495 17023 leader_election.cc:288] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 576bfcf156164a67aae5c49b0e21c1ec (127.3.152.193:35147), bad6d380d13c43ee9773b9afe47aab38 (127.3.152.195:45399)
W20260925 10:15:37.778370 16718 tablet.cc:2369] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:15:37.778976 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.774935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:54783 (local address 127.3.152.195:45399)
0925 10:15:37.774982 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.774984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.775104 (+ 120us) server_negotiation.cc:407] Connection header received
0925 10:15:37.775210 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.775211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.775221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.775250 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:37.775477 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.776252 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778038 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.778552 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778604 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.778614 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.778861 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.778905 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:15:37.779045 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.775022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:48809 (local address 127.3.152.193:35147)
0925 10:15:37.775064 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.775066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.775194 (+ 128us) server_negotiation.cc:407] Connection header received
0925 10:15:37.775291 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.775292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.775301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.775330 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:37.775529 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.776290 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778082 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.778672 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778720 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.778727 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.778938 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.778976 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:15:37.779251 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.774715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:45399 (local address 127.3.152.194:54783)
0925 10:15:37.775040 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.775052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:37.775132 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:37.775321 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:37.775322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:37.775323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:37.775404 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.775405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.776355 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.776355 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.777932 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.777935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.778708 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.778778 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.778782 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:15:37.779112 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 10:15:37.779157 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":101,"threads_started":1}
I20260925 10:15:37.779363 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.774813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35147 (local address 127.3.152.194:48809)
0925 10:15:37.775130 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.775141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:37.775218 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:37.775393 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:37.775394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:37.775396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:37.775462 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.775464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.776390 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.776391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.777963 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:37.777967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.778788 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.778789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:37.778860 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.778863 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:15:37.779244 (+ 381us) client_negotiation.cc:240] Negotiation successful
0925 10:15:37.779287 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":198,"thread_start_us":85,"threads_started":1}
I20260925 10:15:37.779628 16929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a9a89d678a4df9a0f4c974852becc8" candidate_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad6d380d13c43ee9773b9afe47aab38" is_pre_election: true
I20260925 10:15:37.779742 16671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a9a89d678a4df9a0f4c974852becc8" candidate_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576bfcf156164a67aae5c49b0e21c1ec" is_pre_election: true
I20260925 10:15:37.779804 16929 raft_consensus.cc:2454] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b794c9d9d0d347a193f154f8dac2bbf9 in term 0.
I20260925 10:15:37.779877 16671 raft_consensus.cc:2454] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b794c9d9d0d347a193f154f8dac2bbf9 in term 0.
I20260925 10:15:37.780055 16738 leader_election.cc:302] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b794c9d9d0d347a193f154f8dac2bbf9, bad6d380d13c43ee9773b9afe47aab38; no voters:
I20260925 10:15:37.780177 17023 raft_consensus.cc:2781] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:15:37.780230 17023 raft_consensus.cc:482] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:15:37.780258 17023 raft_consensus.cc:3037] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:37.781147 17023 raft_consensus.cc:504] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.781288 17023 leader_election.cc:288] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 576bfcf156164a67aae5c49b0e21c1ec (127.3.152.193:35147), bad6d380d13c43ee9773b9afe47aab38 (127.3.152.195:45399)
I20260925 10:15:37.781481 16671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a9a89d678a4df9a0f4c974852becc8" candidate_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576bfcf156164a67aae5c49b0e21c1ec"
I20260925 10:15:37.781531 16929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22a9a89d678a4df9a0f4c974852becc8" candidate_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bad6d380d13c43ee9773b9afe47aab38"
I20260925 10:15:37.781548 16671 raft_consensus.cc:3037] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:37.781603 16929 raft_consensus.cc:3037] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:37.782541 16671 raft_consensus.cc:2454] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b794c9d9d0d347a193f154f8dac2bbf9 in term 1.
I20260925 10:15:37.782621 16929 raft_consensus.cc:2454] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b794c9d9d0d347a193f154f8dac2bbf9 in term 1.
I20260925 10:15:37.782730 16739 leader_election.cc:302] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576bfcf156164a67aae5c49b0e21c1ec, b794c9d9d0d347a193f154f8dac2bbf9; no voters:
I20260925 10:15:37.782804 17023 raft_consensus.cc:2781] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:37.782938 17023 raft_consensus.cc:686] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 1 LEADER]: Becoming Leader. State: Replica: b794c9d9d0d347a193f154f8dac2bbf9, State: Running, Role: LEADER
I20260925 10:15:37.783032 17023 consensus_queue.cc:237] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } }
I20260925 10:15:37.783833 16517 catalog_manager.cc:5495] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b794c9d9d0d347a193f154f8dac2bbf9 (127.3.152.194). New cstate: current_term: 1 leader_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b794c9d9d0d347a193f154f8dac2bbf9" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 40901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 } health_report { overall_health: UNKNOWN } } }
I20260925 10:15:37.795727 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.792613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.3.152.194:40901)
0925 10:15:37.792677 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:37.792680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:37.792689 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:15:37.792723 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:37.792724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:37.792743 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:37.792820 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:15:37.793006 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.793741 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.794861 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:37.795133 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:37.795200 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:37.795385 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:15:37.795509 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:15:37.795587 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:37.795633 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 10:15:37.795659 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:15:37.797781 16671 raft_consensus.cc:1261] T 22a9a89d678a4df9a0f4c974852becc8 P 576bfcf156164a67aae5c49b0e21c1ec [term 1 FOLLOWER]: Refusing update from remote peer b794c9d9d0d347a193f154f8dac2bbf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:37.797821 16929 raft_consensus.cc:1261] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 1 FOLLOWER]: Refusing update from remote peer b794c9d9d0d347a193f154f8dac2bbf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:37.798175 17029 consensus_queue.cc:1035] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bad6d380d13c43ee9773b9afe47aab38" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 45399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:37.798316 17023 consensus_queue.cc:1035] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576bfcf156164a67aae5c49b0e21c1ec" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:37.802093 17034 mvcc.cc:205] Tried to move back new op lower bound from 7333197159616778240 to 7333197159559860224. Current Snapshot: MvccSnapshot[applied={T|T < 7333197159616778240}]
I20260925 10:15:37.802812 17033 mvcc.cc:205] Tried to move back new op lower bound from 7333197159616778240 to 7333197159559860224. Current Snapshot: MvccSnapshot[applied={T|T < 7333197159616778240}]
I20260925 10:15:37.805488 16516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:37.805581 16516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:37.806411 16516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c4a57a4d28040a6ba137d356998ca58: Sending DeleteTablet for 3 replicas of tablet 22a9a89d678a4df9a0f4c974852becc8
I20260925 10:15:37.806882 16785 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:37 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:58740
I20260925 10:15:37.807093 16902 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:37 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49978
I20260925 10:15:37.807348 16651 tablet_service.cc:1514] Processing DeleteTablet for tablet 22a9a89d678a4df9a0f4c974852becc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:37 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:48282
I20260925 10:15:37.808050 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16556
I20260925 10:15:37.812402 17042 tablet_replica.cc:330] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: stopping tablet replica
I20260925 10:15:37.813102 17041 tablet_replica.cc:330] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: stopping tablet replica
I20260925 10:15:37.813321 17042 raft_consensus.cc:2229] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:15:37.813488 17041 raft_consensus.cc:2229] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:15:37.813690 17041 raft_consensus.cc:2258] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:37.813858 17042 raft_consensus.cc:2258] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:37.817322 17042 ts_tablet_manager.cc:1905] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:37.817849 17041 ts_tablet_manager.cc:1905] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:37.819556 17041 ts_tablet_manager.cc:1918] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:37.819701 17041 log.cc:1196] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/22a9a89d678a4df9a0f4c974852becc8
I20260925 10:15:37.820063 17041 ts_tablet_manager.cc:1939] T 22a9a89d678a4df9a0f4c974852becc8 P b794c9d9d0d347a193f154f8dac2bbf9: Deleting consensus metadata
I20260925 10:15:37.820565 16502 catalog_manager.cc:4841] TS b794c9d9d0d347a193f154f8dac2bbf9 (127.3.152.194:40901): tablet 22a9a89d678a4df9a0f4c974852becc8 (table test-table [id=135517749f6647c786cfa216c2d5a42d]) successfully deleted
W20260925 10:15:37.820971 16502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:35147 (error 108)
I20260925 10:15:37.821853 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:37.821727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35147 (local address 127.0.0.1:48290)
0925 10:15:37.821775 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:37.821815 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:15:37.822057 16502 catalog_manager.cc:4584] TS 576bfcf156164a67aae5c49b0e21c1ec (127.3.152.193:35147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 22a9a89d678a4df9a0f4c974852becc8: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35147: connect: Connection refused (error 111)
I20260925 10:15:37.822326 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16719
I20260925 10:15:37.822381 17042 ts_tablet_manager.cc:1918] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:37.822449 17042 log.cc:1196] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/22a9a89d678a4df9a0f4c974852becc8
I20260925 10:15:37.822723 17042 ts_tablet_manager.cc:1939] T 22a9a89d678a4df9a0f4c974852becc8 P bad6d380d13c43ee9773b9afe47aab38: Deleting consensus metadata
I20260925 10:15:37.823292 16501 catalog_manager.cc:4841] TS bad6d380d13c43ee9773b9afe47aab38 (127.3.152.195:45399): tablet 22a9a89d678a4df9a0f4c974852becc8 (table test-table [id=135517749f6647c786cfa216c2d5a42d]) successfully deleted
I20260925 10:15:37.830160 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16854
I20260925 10:15:37.837862 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 16484
I20260925 10:15:37.843544 3683 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 10:15:38.188516 3683 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 10:15:38.188675 3683 mini_ranger.cc:66] Stopping Ranger...
I20260925 10:15:38.533449 3683 mini_ranger.cc:68] Stopped Ranger
2026-09-25 10:15:38.533 UTC [15550] LOG: received smart shutdown request
2026-09-25 10:15:38.537 UTC [15550] LOG: background worker "logical replication launcher" (PID 15559) exited with exit code 1
2026-09-25 10:15:38.538 UTC [15554] LOG: shutting down
2026-09-25 10:15:38.717 UTC [15550] LOG: database system is shut down
2026-09-25T10:15:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48446 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:15:38 dist-test-slave-dzwq krb5kdc[17053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:15:38 dist-test-slave-dzwq krb5kdc[17053](info): set up 2 sockets
Sep 25 10:15:38 dist-test-slave-dzwq krb5kdc[17053](info): commencing operation
krb5kdc: starting...
W20260925 10:15:39.956679 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:15:40.083637 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36141
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:40.164631 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:40.164844 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:40.164866 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:40.166018 17066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:40.166060 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:40.166074 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:40.166087 17066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:40.166159 17066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:40.167691 17066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:40.167943 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:40.168231 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:40.173162 17074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:40.173167 17072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:40.173240 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:40.173406 17066 server_base.cc:1034] running on GCE node
I20260925 10:15:40.173723 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:40.174023 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:40.175174 17066 hybrid_clock.cc:648] HybridClock initialized: now 1790331340175154 us; error 36 us; skew 500 ppm
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:40.181686 17066 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:40.182219 17066 webserver.cc:460] Webserver started at http://127.3.152.254:38405/ using document root <none> and password file <none>
I20260925 10:15:40.182471 17066 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:40.182542 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:40.182698 17066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:40.183902 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "9fd1aee38df8459b9293b1eb931a102f"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "4b1e3e05bd101e7c8b1b3fab3ab1e42f"
server_key_iv: "e82155d150cac9e0c2fc48f50e642570"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.184337 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "9fd1aee38df8459b9293b1eb931a102f"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "4b1e3e05bd101e7c8b1b3fab3ab1e42f"
server_key_iv: "e82155d150cac9e0c2fc48f50e642570"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.185787 17066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:40.186661 17081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:40.186841 17066 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:40.186944 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "9fd1aee38df8459b9293b1eb931a102f"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "4b1e3e05bd101e7c8b1b3fab3ab1e42f"
server_key_iv: "e82155d150cac9e0c2fc48f50e642570"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.187029 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:40.198240 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:40.200749 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:40.200941 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:40.205313 17066 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:36141
I20260925 10:15:40.205351 17133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:36141 every 8 connection(s)
I20260925 10:15:40.205732 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:40.206631 17134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:40.208230 17134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.208784 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17066
I20260925 10:15:40.208870 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:15:40.209121 3683 external_mini_cluster.cc:1428] Setting key 6134142f973a3456a1311581109bce05
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.253@KRBTEST.COM; defaulting to no policy
I20260925 10:15:40.219732 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.209076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.3.152.254:36141)
0925 10:15:40.209340 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.209346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.209389 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:15:40.209969 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.209971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.209994 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.210106 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.210856 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.211917 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.212910 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.213455 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.213588 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.216006 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.216014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.216015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.216023 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.217959 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.218351 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.218353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.218353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.218433 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.218665 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.218666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.218667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.218944 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.219043 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.219244 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.219451 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
I20260925 10:15:40.219731 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.209044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36141 (local address 127.0.0.1:60962)
0925 10:15:40.209232 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.209291 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.209804 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.210140 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.210142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.210489 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.210741 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.210748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.212075 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.212076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.212826 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.212828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.213551 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.213552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.213624 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.214239 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.214249 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.215893 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.218039 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.218046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.218048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.218270 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.218502 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.218502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.218503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.218588 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.219022 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.219027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.219114 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.219218 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.219496 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":124,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.253@KRBTEST.COM" created.
I20260925 10:15:40.220635 17134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.221040 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.220921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47664)
0925 10:15:40.220974 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.221017 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:15:40.221230 17083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
I20260925 10:15:40.221691 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.221592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47668)
0925 10:15:40.221643 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.221672 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:15:40.221915 17134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:39543: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
I20260925 10:15:40.236464 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:39543
--webserver_interface=127.3.152.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:40.271169 17134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } attempt: 1
I20260925 10:15:40.271693 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.271554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47680)
0925 10:15:40.271619 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.271661 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:15:40.272148 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.272036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47696)
0925 10:15:40.272081 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.272116 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 10:15:40.272245 17134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:39543: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
W20260925 10:15:40.320770 17141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:40.321012 17141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:40.321038 17141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:40.322218 17141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:40.322269 17141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:40.322286 17141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:40.322300 17141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:40.322314 17141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:40.323899 17141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:39543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.152.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:40.324174 17141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:40.324460 17141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:40.329808 17149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:40.329891 17141 server_base.cc:1034] running on GCE node
W20260925 10:15:40.329813 17147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:40.329830 17146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:40.330317 17141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:40.330618 17141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:40.331776 17141 hybrid_clock.cc:648] HybridClock initialized: now 1790331340331750 us; error 35 us; skew 500 ppm
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:40.334622 17141 init.cc:377] Logged in from keytab as kudu/127.3.152.253@KRBTEST.COM (short username kudu)
I20260925 10:15:40.335153 17141 webserver.cc:460] Webserver started at http://127.3.152.253:41175/ using document root <none> and password file <none>
I20260925 10:15:40.335417 17141 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:40.335489 17141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:40.335639 17141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:40.336796 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data/instance:
uuid: "abc703d2c1854dc4928e3b3b923afb60"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "5e4bec5c456718983a415c2adafc60b4"
server_key_iv: "28b7ca571b115c0a51b8a2c6dd4c2e28"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.337157 17141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal/instance:
uuid: "abc703d2c1854dc4928e3b3b923afb60"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "5e4bec5c456718983a415c2adafc60b4"
server_key_iv: "28b7ca571b115c0a51b8a2c6dd4c2e28"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.338543 17141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:15:40.339377 17156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:40.339587 17141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:40.339715 17141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "abc703d2c1854dc4928e3b3b923afb60"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "5e4bec5c456718983a415c2adafc60b4"
server_key_iv: "28b7ca571b115c0a51b8a2c6dd4c2e28"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.339797 17141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:40.340378 17134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } attempt: 2
I20260925 10:15:40.340809 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.340645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47710)
0925 10:15:40.340740 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.340781 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 10:15:40.341130 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.341031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47724)
0925 10:15:40.341076 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.341100 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:15:40.341236 17134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:39543: Network error: Client connection negotiation failed: client connection to 127.3.152.253:39543: connect: Connection refused (error 111)
I20260925 10:15:40.359678 17141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:40.361836 17141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:40.362006 17141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:40.366297 17141 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.253:39543
I20260925 10:15:40.366340 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.253:39543 every 8 connection(s)
I20260925 10:15:40.366710 17141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
I20260925 10:15:40.367537 17209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:40.369268 17209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.371840 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17141
I20260925 10:15:40.371961 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-1/wal/instance
I20260925 10:15:40.372189 3683 external_mini_cluster.cc:1428] Setting key 7461c6766f4d32b2106b7600f0d64a9e
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:40.378993 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.370076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.3.152.254:36141)
0925 10:15:40.370159 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.370161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.370376 (+ 215us) server_negotiation.cc:407] Connection header received
0925 10:15:40.370853 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.370855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.370869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.370937 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.371602 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.372268 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.373287 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.373581 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.373647 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.376068 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.376070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.376070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.376079 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.377543 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.377977 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.377979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.377979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.378043 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.378326 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.378327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.378327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.378448 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.378520 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.378811 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.378899 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.3.152.252@KRBTEST.COM; defaulting to no policy
I20260925 10:15:40.379185 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.370034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36141 (local address 127.0.0.1:60968)
0925 10:15:40.370211 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.370256 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.370740 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.371017 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.371020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.371294 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.371495 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.371499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.372369 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.372372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.373164 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.373166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.373676 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.373677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.373843 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.374352 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.374362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.375918 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.377655 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.377663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.377666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.377898 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.378133 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.378134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.378134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.378250 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.378522 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.378530 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.378675 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.378778 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.378946 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260925 10:15:40.379779 17209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.252@KRBTEST.COM" created.
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:15:40.387820 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.380309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.3.152.253:39543)
0925 10:15:40.380432 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.380435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.380451 (+ 16us) server_negotiation.cc:407] Connection header received
0925 10:15:40.380466 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.380468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.380478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.380558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.380890 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.381663 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.382615 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.382917 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.382979 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.384866 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.384873 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.384874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.384880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.386559 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.387033 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.387035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.387036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.387091 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.387297 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.387298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.387298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.387426 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.387492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.387674 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.387742 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260925 10:15:40.387980 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.380101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47738)
0925 10:15:40.380153 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.380165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.380206 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.380558 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.380559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.380756 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.380805 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.380806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.381768 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.381769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.382529 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.382532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.382997 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.382998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.383081 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.383553 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.383557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.384761 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.386702 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.386703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.386704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.386941 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.387155 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.387156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.387157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.387229 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.387501 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.387501 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.387550 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.387649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.387811 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":57}
I20260925 10:15:40.388739 17209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.389112 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.388976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:38982)
0925 10:15:40.389029 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.389079 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:15:40.389262 17160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
I20260925 10:15:40.389801 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.389665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:38984)
0925 10:15:40.389741 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.389771 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 10:15:40.389914 17209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:39003: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
I20260925 10:15:40.399703 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:39003
--webserver_interface=127.3.152.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:40.432422 17134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } attempt: 3
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:15:40.439172 17209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } attempt: 1
I20260925 10:15:40.439596 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.439448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:38988)
0925 10:15:40.439503 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.439538 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:15:40.440075 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.439953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:38998)
0925 10:15:40.440014 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.440046 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:15:40.440152 17209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:39003: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
I20260925 10:15:40.440357 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.432850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47748)
0925 10:15:40.432910 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.432929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.432992 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.433178 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.433179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.433357 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.433421 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.433424 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.434190 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.434192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.434738 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.434740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.435141 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.435142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.435229 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.435786 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.435791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.437420 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.439150 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.439152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.439152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.439393 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.439601 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.439602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.439603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.439680 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.440028 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.440029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.440091 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.440176 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.440286 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:15:40.440642 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.432938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.3.152.253:39543)
0925 10:15:40.433012 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.433014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.433025 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:15:40.433053 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.433054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.433063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.433168 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.433515 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.434058 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.434830 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.435044 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.435130 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.437549 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.437551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.437551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.437558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.439013 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.439454 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.439456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.439456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.439519 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.439764 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.439765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.439765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.439911 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.440034 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.440505 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.440591 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 10:15:40.441296 17134 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.441642 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.441542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39006)
0925 10:15:40.441588 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.441616 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 10:15:40.441947 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.441857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39010)
0925 10:15:40.441905 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.441927 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:15:40.442039 17134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:39003: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
I20260925 10:15:40.473210 17134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } attempt: 1
I20260925 10:15:40.473685 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.473540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39024)
0925 10:15:40.473615 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.473656 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 10:15:40.474079 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.473989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39030)
0925 10:15:40.474038 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.474062 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:15:40.474188 17134 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:39003: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
W20260925 10:15:40.484694 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:40.484911 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:40.484934 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:40.486102 17216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:40.486147 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:40.486162 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:40.486176 17216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:40.486187 17216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:40.487761 17216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:36141,127.3.152.253:39543,127.3.152.252:39003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.152.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:40.488044 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:40.488296 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:40.493306 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:40.493314 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:40.493366 17216 server_base.cc:1034] running on GCE node
W20260925 10:15:40.493314 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:40.493726 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:40.493994 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:40.495173 17216 hybrid_clock.cc:648] HybridClock initialized: now 1790331340495183 us; error 54 us; skew 500 ppm
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:40.497967 17216 init.cc:377] Logged in from keytab as kudu/127.3.152.252@KRBTEST.COM (short username kudu)
I20260925 10:15:40.498483 17216 webserver.cc:460] Webserver started at http://127.3.152.252:34427/ using document root <none> and password file <none>
I20260925 10:15:40.498759 17216 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:40.498834 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:40.498996 17216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:40.500314 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data/instance:
uuid: "652a447448394975b94b9712f54e1aa2"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "28b7c5cb918cd9d392f435fc1c01e21f"
server_key_iv: "8c98ae3263488a3bcdbccc6af0f0f4c2"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.500723 17216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal/instance:
uuid: "652a447448394975b94b9712f54e1aa2"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "28b7c5cb918cd9d392f435fc1c01e21f"
server_key_iv: "8c98ae3263488a3bcdbccc6af0f0f4c2"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.502334 17216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:15:40.503144 17231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:40.503336 17216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:40.503449 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "652a447448394975b94b9712f54e1aa2"
format_stamp: "Formatted at 2026-09-25 10:15:40 on dist-test-slave-dzwq"
server_key: "28b7c5cb918cd9d392f435fc1c01e21f"
server_key_iv: "8c98ae3263488a3bcdbccc6af0f0f4c2"
server_key_version: "encryptionkey@0"
I20260925 10:15:40.503539 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:40.508279 17209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } attempt: 2
I20260925 10:15:40.508692 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.508559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39044)
0925 10:15:40.508617 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.508652 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:15:40.509068 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.508973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39046)
0925 10:15:40.509018 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.509042 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 10:15:40.509161 17209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:39003: Network error: Client connection negotiation failed: client connection to 127.3.152.252:39003: connect: Connection refused (error 111)
I20260925 10:15:40.530705 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:40.532917 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:40.533106 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:40.537251 17216 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.252:39003
I20260925 10:15:40.537321 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.252:39003 every 8 connection(s)
I20260925 10:15:40.537708 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
I20260925 10:15:40.538661 17284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:40.540297 17284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.545104 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17216
I20260925 10:15:40.545238 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790331159020724-3683-0/minicluster-data/master-2/wal/instance
I20260925 10:15:40.545514 3683 external_mini_cluster.cc:1428] Setting key 029defe1bba6f3f9b8de1fd6362bc835
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:40.549340 17134 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } attempt: 2
I20260925 10:15:40.551056 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.541074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.3.152.254:36141)
0925 10:15:40.541153 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.541156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.541381 (+ 225us) server_negotiation.cc:407] Connection header received
0925 10:15:40.541836 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.541837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.541848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.541910 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.542622 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.543181 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.544487 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.544756 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.544971 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.547516 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.547518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.547518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.547525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.549197 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.549806 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.549807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.549808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.549873 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.550272 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.550273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.550274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.550413 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.550469 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.550842 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.550927 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:15:40.551226 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.541056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:36141 (local address 127.0.0.1:60980)
0925 10:15:40.541256 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.541293 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.541742 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.542004 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.542007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.542339 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.542560 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.542565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.543302 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.543303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.544398 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.544401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.544988 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.544989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.545111 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.545837 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.545849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.547414 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.549301 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.549326 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.549329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.549694 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.549978 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.549979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.549979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.550139 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.550486 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.550493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.550667 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.550800 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.550975 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":118,"threads_started":1}
I20260925 10:15:40.551750 17284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:15:40.556466 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.545953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.3.152.252:39003)
0925 10:15:40.546159 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.546165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.546186 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:15:40.546268 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.546271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.546293 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.546395 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.547097 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.547910 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.549066 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.549586 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.549661 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.552746 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.552789 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.552790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.552797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.554893 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.555323 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.555327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.555327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.555404 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.555663 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.555663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.555664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.555822 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.555998 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.556243 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.556324 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:15:40.559353 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.549657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39072)
0925 10:15:40.549710 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.549721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.549753 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.550004 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.550005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.550175 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.550226 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.550228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.551421 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.551422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.552312 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.552315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.552860 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.552861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.552920 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.553298 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.553303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.554800 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.557415 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.557416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.557416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.557586 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.557878 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.557878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.557879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.557933 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.559070 (+ 1137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.559070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.559112 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.559174 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.559260 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:15:40.559665 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.549721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.3.152.252:39003)
0925 10:15:40.549873 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.549875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.549881 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:15:40.549899 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.549900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.549910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.549981 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.550294 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.551312 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.552407 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.552764 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.552850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.554901 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.554902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.554903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.554909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.557336 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.557757 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.557758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.557759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.557814 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.558278 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.558279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.558279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.558435 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.558521 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.559472 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.559549 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260925 10:15:40.560235 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.552032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:39543 (local address 127.0.0.1:47756)
0925 10:15:40.552073 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.552084 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.552130 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.552346 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.552346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.552508 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.552561 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.552562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.553427 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.553428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.554148 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.554151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.554643 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.554644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.554700 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.555359 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.555371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.557005 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.559010 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.559011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.559012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.559195 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.559417 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.559418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.559418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.559490 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.559906 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.559907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.559949 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.560041 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.560140 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 10:15:40.560272 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.552147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.3.152.253:39543)
0925 10:15:40.552191 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.552192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.552200 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:15:40.552225 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.552225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.552236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.552297 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.552654 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.553358 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.554258 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.554485 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.554580 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.557079 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.557080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.557081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.557089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.558905 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.559273 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.559275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.559275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.559333 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.559579 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.559580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.559580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.559712 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.559765 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.560097 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.560191 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:15:40.560638 17284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:15:40.561889 17134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: Bootstrap starting.
I20260925 10:15:40.562691 17134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: Neither blocks nor log segments found. Creating new log.
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:40.563023 17134 log.cc:824] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:40.564430 17134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: No bootstrap required, opened a new log
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:15:40.566161 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.557719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.3.152.254:36141)
0925 10:15:40.557755 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.557757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.557825 (+ 68us) server_negotiation.cc:407] Connection header received
0925 10:15:40.557931 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.557932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.557943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.557993 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.558526 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.558969 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.560023 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.560334 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.560408 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.563237 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.563238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.563239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.563245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.564715 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.565362 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.565363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.565363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.565419 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.565619 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.565620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.565621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.565765 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.565924 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.565966 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.566048 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:15:40.566900 17134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.567063 17134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:40.567111 17134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd1aee38df8459b9293b1eb931a102f, State: Initialized, Role: FOLLOWER
I20260925 10:15:40.567245 17134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.567621 17134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:40.568023 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.568114 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:15:40.568943 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.560898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39082)
0925 10:15:40.561198 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.561207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.561277 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.561412 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.561412 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.561553 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.561581 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.561582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.562327 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.562329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.562923 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.562925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.563437 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.563438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.563503 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.563819 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.563825 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.565246 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.567565 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.567566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.567566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.567802 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.568092 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.568092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.568093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.568154 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.568609 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.568610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.568657 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.568728 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.568821 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286}
I20260925 10:15:40.569187 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.560929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.3.152.252:39003)
0925 10:15:40.560965 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.560967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.561261 (+ 294us) server_negotiation.cc:407] Connection header received
0925 10:15:40.561339 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.561339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.561349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.561408 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.561666 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.562245 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.563006 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.563346 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.563426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.565330 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.565331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.565331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.565339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.567472 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.567894 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.567895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.567895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.567958 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.568290 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.568290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.568291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.568507 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.568617 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.569052 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.569114 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 10:15:40.569942 17302 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:15:40.569989 17302 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:15:40.570636 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: Bootstrap starting.
I20260925 10:15:40.571384 17284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:40.571704 17284 log.cc:824] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:40.572510 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: No bootstrap required, opened a new log
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:15:40.570744 17066 master_runner.cc:186] Error getting master registration for 127.3.152.253:39543: OK, instance_id { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" instance_seqno: 1790331340365122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:15:40.573987 17066 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:15:40.574678 17284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.574848 17284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:40.574887 17284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 652a447448394975b94b9712f54e1aa2, State: Initialized, Role: FOLLOWER
I20260925 10:15:40.575032 17284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.575310 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.575366 17284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:40.577121 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:15:40.577468 17316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:15:40.577526 17316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:15:40.577898 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.567055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.3.152.253:39543)
0925 10:15:40.567097 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.567099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.567177 (+ 78us) server_negotiation.cc:407] Connection header received
0925 10:15:40.567276 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.567277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.567287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.567344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.567906 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.568641 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.569930 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.570258 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.570331 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.573711 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.573713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.573713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.573723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.575439 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.575876 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.575877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.575878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.575942 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.576206 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.576208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.576208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.576381 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.577644 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.577695 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.577789 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:15:40.579504 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:15:40.579309 17216 master_runner.cc:186] Error getting master registration for 127.3.152.253:39543: OK, instance_id { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" instance_seqno: 1790331340365122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:15:40.581629 17216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:15:40.591207 17213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.582531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.3.152.253:39543)
0925 10:15:40.582583 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.582585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.583547 (+ 962us) server_negotiation.cc:407] Connection header received
0925 10:15:40.583609 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.583610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.583621 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.583675 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.584448 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.585198 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.586170 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.586522 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.586608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.588030 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.588032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.588032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.588039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.589811 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590362 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590419 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590608 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590730 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.590800 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.591017 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.591095 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:15:40.591284 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.581012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.3.152.254:36141)
0925 10:15:40.581059 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.581062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.581344 (+ 282us) server_negotiation.cc:407] Connection header received
0925 10:15:40.581457 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.581458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.581469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.581504 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.582773 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.583323 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.585288 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.585639 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.585691 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.587709 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.587711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.587711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.587720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.589712 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590244 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590309 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590538 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590710 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.590806 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.591085 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.591169 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:15:40.591457 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.581034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.3.152.252:39003)
0925 10:15:40.581072 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.581075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.581343 (+ 268us) server_negotiation.cc:407] Connection header received
0925 10:15:40.581456 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.581457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.581469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.581504 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.582544 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.583298 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.584353 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.584667 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.584734 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.586772 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.586774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.586774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.586782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.589963 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590490 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590550 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.590771 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.590772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.590772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.590929 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.591035 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.591263 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.591345 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:15:40.600307 17209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } attempt: 3
Sep 25 10:15:40 dist-test-slave-dzwq krb5kdc[17053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:15:40.609288 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.600697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.3.152.252:39003)
0925 10:15:40.600844 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:40.600846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:40.600853 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:15:40.600876 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:40.600877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:40.600887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:40.600943 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:40.601310 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.602194 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.603193 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.603531 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.603609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.606128 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:40.606130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:40.606130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:40.606138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:40.608006 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.608433 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.608434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.608435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.608497 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:40.608721 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:40.608722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:40.608722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:40.608859 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:40.608928 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:40.609099 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 10:15:40.609186 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260925 10:15:40.609303 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:40.600614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:39003 (local address 127.0.0.1:39102)
0925 10:15:40.600696 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:40.600710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:40.600763 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:40.600969 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:40.600970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:40.601169 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:40.601225 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.601227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.602288 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.602289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.603098 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:40.603100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:40.603614 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:40.603615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:40.603674 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:40.604293 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:40.604299 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:40.606015 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:40.608097 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.608098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.608099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.608349 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.608574 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:40.608575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:40.608575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:40.608650 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:40.608932 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:40.608933 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:40.608986 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 10:15:40.609073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 10:15:40.609188 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 10:15:40.611541 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: Bootstrap starting.
I20260925 10:15:40.612406 17209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:40.612756 17209 log.cc:824] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:40.613696 17209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: No bootstrap required, opened a new log
I20260925 10:15:40.615947 17209 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.616151 17209 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:40.616200 17209 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc703d2c1854dc4928e3b3b923afb60, State: Initialized, Role: FOLLOWER
I20260925 10:15:40.616330 17209 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.616686 17209 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:40.616622 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.616853 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:15:40.618753 17337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:15:40.618886 17337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:15:40.621430 17141 master_runner.cc:263] No leader master found from master abc703d2c1854dc4928e3b3b923afb60
I20260925 10:15:40.621495 17141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:15:40.647452 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:40.647603 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.647897 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fd1aee38df8459b9293b1eb931a102f (127.3.152.254:36141), abc703d2c1854dc4928e3b3b923afb60 (127.3.152.253:39543)
I20260925 10:15:40.648435 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "652a447448394975b94b9712f54e1aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd1aee38df8459b9293b1eb931a102f" is_pre_election: true
I20260925 10:15:40.648607 17108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 652a447448394975b94b9712f54e1aa2 in term 0.
I20260925 10:15:40.648806 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a447448394975b94b9712f54e1aa2, 9fd1aee38df8459b9293b1eb931a102f; no voters:
I20260925 10:15:40.649825 17305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:15:40.649895 17305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:15:40.649919 17305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:40.650779 17305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.650890 17305 leader_election.cc:288] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [CANDIDATE]: Term 1 election: Requested vote from peers 9fd1aee38df8459b9293b1eb931a102f (127.3.152.254:36141), abc703d2c1854dc4928e3b3b923afb60 (127.3.152.253:39543)
I20260925 10:15:40.651286 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "652a447448394975b94b9712f54e1aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fd1aee38df8459b9293b1eb931a102f"
I20260925 10:15:40.651358 17108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:40.652343 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "652a447448394975b94b9712f54e1aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc703d2c1854dc4928e3b3b923afb60"
I20260925 10:15:40.652463 17182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:40.652359 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "652a447448394975b94b9712f54e1aa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc703d2c1854dc4928e3b3b923afb60" is_pre_election: true
I20260925 10:15:40.653041 17108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a447448394975b94b9712f54e1aa2 in term 1.
I20260925 10:15:40.653239 17234 leader_election.cc:302] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 652a447448394975b94b9712f54e1aa2, 9fd1aee38df8459b9293b1eb931a102f; no voters:
I20260925 10:15:40.653333 17305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:40.653424 17182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 652a447448394975b94b9712f54e1aa2 in term 1.
I20260925 10:15:40.653496 17305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 652a447448394975b94b9712f54e1aa2, State: Running, Role: LEADER
I20260925 10:15:40.653627 17305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } }
I20260925 10:15:40.654038 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652a447448394975b94b9712f54e1aa2. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.654119 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:40.656129 17341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:40.656347 17341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:40.660940 17108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [term 1 FOLLOWER]: Refusing update from remote peer 652a447448394975b94b9712f54e1aa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:40.661250 17182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [term 1 FOLLOWER]: Refusing update from remote peer 652a447448394975b94b9712f54e1aa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:40.661839 17305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:40.661989 17305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:40.663337 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652a447448394975b94b9712f54e1aa2. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.663439 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:15:40.664582 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 652a447448394975b94b9712f54e1aa2. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.664681 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:15:40.666559 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.666648 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:40.667102 17326 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.667174 17326 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:15:40.667404 17342 mvcc.cc:205] Tried to move back new op lower bound from 7333197171344617472 to 7333197171318128640. Current Snapshot: MvccSnapshot[applied={T|T < 7333197171344617472}]
I20260925 10:15:40.667827 17344 mvcc.cc:205] Tried to move back new op lower bound from 7333197171344617472 to 7333197171318128640. Current Snapshot: MvccSnapshot[applied={T|T < 7333197171344617472}]
I20260925 10:15:40.668867 17339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.668962 17339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:40.669485 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "652a447448394975b94b9712f54e1aa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd1aee38df8459b9293b1eb931a102f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36141 } } peers { permanent_uuid: "abc703d2c1854dc4928e3b3b923afb60" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 39543 } } peers { permanent_uuid: "652a447448394975b94b9712f54e1aa2" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 39003 } } }
I20260925 10:15:40.669557 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:15:40.670678 17341 catalog_manager.cc:1347] Generated new cluster ID: 55a67e44d9714ed49672d6c8e09690fe
I20260925 10:15:40.670724 17341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:15:40.675506 17346 mvcc.cc:205] Tried to move back new op lower bound from 7333197171344617472 to 7333197171318128640. Current Snapshot: MvccSnapshot[applied={T|T < 7333197171344617472}]
I20260925 10:15:40.683748 17341 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:15:40.684576 17341 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:40.695431 17341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 652a447448394975b94b9712f54e1aa2: Generated new TSK 0
I20260925 10:15:40.695716 17341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:15:41.570479 17302 catalog_manager.cc:1259] Loaded cluster ID: 55a67e44d9714ed49672d6c8e09690fe
I20260925 10:15:41.570585 17302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: loading cluster ID for follower catalog manager: success
I20260925 10:15:41.571657 17302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: acquiring CA information for follower catalog manager: success
I20260925 10:15:41.572072 17302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fd1aee38df8459b9293b1eb931a102f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:15:41.619395 17337 catalog_manager.cc:1259] Loaded cluster ID: 55a67e44d9714ed49672d6c8e09690fe
I20260925 10:15:41.619501 17337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: loading cluster ID for follower catalog manager: success
I20260925 10:15:41.620358 17337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: acquiring CA information for follower catalog manager: success
I20260925 10:15:41.620702 17337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P abc703d2c1854dc4928e3b3b923afb60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:15:41.758091 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17066
I20260925 10:15:41.763356 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17141
I20260925 10:15:41.768092 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17216
2026-09-25T10:15:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2929 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:15:41 dist-test-slave-dzwq krb5kdc[17357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:15:41 dist-test-slave-dzwq krb5kdc[17357](info): set up 2 sockets
Sep 25 10:15:41 dist-test-slave-dzwq krb5kdc[17357](info): commencing operation
krb5kdc: starting...
W20260925 10:15:42.808729 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:15:42 dist-test-slave-dzwq krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:15:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:15:42.926709 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43033
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:43033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:43.009649 17370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:43.009883 17370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:43.009907 17370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:43.011209 17370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:43.011253 17370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:43.011268 17370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:43.011281 17370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:43.011293 17370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:43.012887 17370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:43033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:43.013155 17370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:43.013405 17370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:43.018354 17376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:43.018352 17378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:43.018486 17370 server_base.cc:1034] running on GCE node
W20260925 10:15:43.018355 17375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:43.018850 17370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:43.019187 17370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:43.020325 17370 hybrid_clock.cc:648] HybridClock initialized: now 1790331343020308 us; error 35 us; skew 500 ppm
Sep 25 10:15:43 dist-test-slave-dzwq krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:43.022933 17370 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:43.023415 17370 webserver.cc:460] Webserver started at http://127.3.152.254:43835/ using document root <none> and password file <none>
I20260925 10:15:43.023675 17370 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:43.023744 17370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:43.023895 17370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:43.025003 17370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "100ed8f159bb411ebe384b40b0651848"
format_stamp: "Formatted at 2026-09-25 10:15:43 on dist-test-slave-dzwq"
server_key: "fbea1c69db25fbddf042632f1f900f7a"
server_key_iv: "113a8a4a091e38b4a6d08e71045e0e6d"
server_key_version: "encryptionkey@0"
I20260925 10:15:43.025386 17370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "100ed8f159bb411ebe384b40b0651848"
format_stamp: "Formatted at 2026-09-25 10:15:43 on dist-test-slave-dzwq"
server_key: "fbea1c69db25fbddf042632f1f900f7a"
server_key_iv: "113a8a4a091e38b4a6d08e71045e0e6d"
server_key_version: "encryptionkey@0"
I20260925 10:15:43.026747 17370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:15:43.027472 17385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:43.027644 17370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:15:43.027771 17370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "100ed8f159bb411ebe384b40b0651848"
format_stamp: "Formatted at 2026-09-25 10:15:43 on dist-test-slave-dzwq"
server_key: "fbea1c69db25fbddf042632f1f900f7a"
server_key_iv: "113a8a4a091e38b4a6d08e71045e0e6d"
server_key_version: "encryptionkey@0"
I20260925 10:15:43.027848 17370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:43.042716 17370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:43.044795 17370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:43.044970 17370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:43.049284 17370 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:43033
I20260925 10:15:43.049328 17437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:43033 every 8 connection(s)
I20260925 10:15:43.049708 17370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:43.050621 17438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:43.051837 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17370
I20260925 10:15:43.051978 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:15:43.052237 3683 external_mini_cluster.cc:1428] Setting key d1c03643f10fd1f7da68490535ba2550
I20260925 10:15:43.053368 17438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: Bootstrap starting.
I20260925 10:15:43.054124 17438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:43.054484 17438 log.cc:824] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:43.055222 17438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: No bootstrap required, opened a new log
I20260925 10:15:43.057655 17438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } }
I20260925 10:15:43.057833 17438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:43.057873 17438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100ed8f159bb411ebe384b40b0651848, State: Initialized, Role: FOLLOWER
I20260925 10:15:43.058032 17438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } }
I20260925 10:15:43.058133 17438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:43.058171 17438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:43.058228 17438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:43.059059 17438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } }
I20260925 10:15:43.059190 17438 leader_election.cc:302] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 100ed8f159bb411ebe384b40b0651848; no voters:
I20260925 10:15:43.059389 17438 leader_election.cc:288] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:15:43.059415 17443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:43.059530 17443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [term 1 LEADER]: Becoming Leader. State: Replica: 100ed8f159bb411ebe384b40b0651848, State: Running, Role: LEADER
I20260925 10:15:43.059669 17438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:43.059684 17443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } }
Sep 25 10:15:43 dist-test-slave-dzwq krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:43.060092 17443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 100ed8f159bb411ebe384b40b0651848. Latest consensus state: current_term: 1 leader_uuid: "100ed8f159bb411ebe384b40b0651848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } } }
I20260925 10:15:43.060204 17443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:43.060110 17444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "100ed8f159bb411ebe384b40b0651848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "100ed8f159bb411ebe384b40b0651848" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43033 } } }
I20260925 10:15:43.060360 17444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848 [sys.catalog]: This master's current role is: LEADER
W20260925 10:15:43.061661 17458 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:15:43.061764 17458 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:15:43.061880 17450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:43.062065 17450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:43.063483 17450 catalog_manager.cc:1347] Generated new cluster ID: 0a16c1dc930c4a01b8d67e4fbdfc8817
I20260925 10:15:43.063534 17450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:15:43.064941 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:43.052892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.3.152.254:43033)
0925 10:15:43.053059 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:43.053068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:43.053095 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:15:43.053559 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:43.053566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:43.053619 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:43.053741 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:43.054664 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:43.055694 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:43.056822 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:43.057231 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:43.057335 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:43.060506 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:43.060510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:43.060511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:43.060522 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:43.062816 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:43.063265 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:43.063268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:43.063268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:43.063370 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:43.063581 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:43.063582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:43.063582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:43.063881 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:43.064518 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:43.064592 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:15:43.064738 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 10:15:43.065898 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:15:43.090265 17450 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:15:43.091099 17450 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:43.095274 17450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 100ed8f159bb411ebe384b40b0651848: Generated new TSK 0
I20260925 10:15:43.095486 17450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:15:43.120352 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17370
2026-09-25T10:15:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1350 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:15:43 dist-test-slave-dzwq krb5kdc[17465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:15:43 dist-test-slave-dzwq krb5kdc[17465](info): set up 2 sockets
Sep 25 10:15:43 dist-test-slave-dzwq krb5kdc[17465](info): commencing operation
krb5kdc: starting...
W20260925 10:15:44.172786 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:15:44.298648 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:44.380735 17478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:44.380946 17478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:44.380970 17478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:44.382148 17478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:44.382189 17478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:44.382203 17478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:44.382223 17478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:44.382279 17478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:44.383803 17478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:44.384096 17478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:44.384337 17478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:44.389202 17483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.389227 17484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.389231 17478 server_base.cc:1034] running on GCE node
W20260925 10:15:44.389186 17486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.389802 17478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:44.390065 17478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:44.391202 17478 hybrid_clock.cc:648] HybridClock initialized: now 1790331344391188 us; error 26 us; skew 500 ppm
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:44.393937 17478 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:44.394434 17478 webserver.cc:460] Webserver started at http://127.3.152.254:41701/ using document root <none> and password file <none>
I20260925 10:15:44.394665 17478 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:44.394716 17478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:44.394874 17478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:44.395953 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "902a5c75dd4946b58fb6b7c08915180b"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "00185af981683066a415f5593852dd4e"
server_key_iv: "a99ccfc6c53c764fe5a7bbc17837a4f0"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.396381 17478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "902a5c75dd4946b58fb6b7c08915180b"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "00185af981683066a415f5593852dd4e"
server_key_iv: "a99ccfc6c53c764fe5a7bbc17837a4f0"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.397774 17478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:44.398538 17493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.398795 17478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:44.398905 17478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "902a5c75dd4946b58fb6b7c08915180b"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "00185af981683066a415f5593852dd4e"
server_key_iv: "a99ccfc6c53c764fe5a7bbc17837a4f0"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.398983 17478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:44.410763 17478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:44.412838 17478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:44.413010 17478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:44.417173 17478 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:38901
I20260925 10:15:44.417215 17545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:38901 every 8 connection(s)
I20260925 10:15:44.417603 17478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:44.418603 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:44.420957 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap starting.
I20260925 10:15:44.421558 17546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:44.421860 17546 log.cc:824] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:44.422551 17546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: No bootstrap required, opened a new log
I20260925 10:15:44.423789 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17478
I20260925 10:15:44.423916 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:15:44.424149 3683 external_mini_cluster.cc:1428] Setting key 2a3270d3ab421a4c8e3fdf731278f764
I20260925 10:15:44.424463 17546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:44.424660 17546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:44.424743 17546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Initialized, Role: FOLLOWER
I20260925 10:15:44.424899 17546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:44.424995 17546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:44.425031 17546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:44.425099 17546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:44.425894 17546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:44.426016 17546 leader_election.cc:302] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902a5c75dd4946b58fb6b7c08915180b; no voters:
I20260925 10:15:44.426193 17546 leader_election.cc:288] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:15:44.426213 17551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:44.426358 17551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 LEADER]: Becoming Leader. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Running, Role: LEADER
I20260925 10:15:44.426483 17546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:44.426498 17551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:44.426959 17551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902a5c75dd4946b58fb6b7c08915180b. Latest consensus state: current_term: 1 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:44.426990 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:44.427089 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:44.427078 17551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:44.427412 17559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:44.428062 17559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:44.429934 17559 catalog_manager.cc:1347] Generated new cluster ID: 110cb4d5ba8347708d503c5260e2e0c1
I20260925 10:15:44.429988 17559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:44.435321 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.424716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.3.152.254:38901)
0925 10:15:44.424891 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.424900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.424967 (+ 67us) server_negotiation.cc:407] Connection header received
0925 10:15:44.425120 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.425128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.425192 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.425332 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:44.426159 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.427633 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.428786 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.429385 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.429499 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.431853 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:44.431858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:44.431858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:44.431871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:44.433703 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.434088 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.434091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.434092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.434211 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.434433 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.434434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.434434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.434714 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:44.434893 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:44.434994 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 10:15:44.435139 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260925 10:15:44.447340 17559 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:15:44.448117 17559 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:44.453003 17559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Generated new TSK 0
I20260925 10:15:44.453194 17559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:15:44.490973 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:44.583495 17572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:44.583720 17572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:44.583746 17572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:44.585018 17572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:44.585067 17572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:44.585125 17572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:15:44.586797 17572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:44.587023 17572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:44.587251 17572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:44.592502 17580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.592519 17578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.592592 17572 server_base.cc:1034] running on GCE node
W20260925 10:15:44.592655 17577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.592854 17572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:44.593056 17572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:44.594198 17572 hybrid_clock.cc:648] HybridClock initialized: now 1790331344594177 us; error 36 us; skew 500 ppm
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:44.596730 17572 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:15:44.597155 17572 webserver.cc:460] Webserver started at http://127.3.152.193:35709/ using document root <none> and password file <none>
I20260925 10:15:44.597414 17572 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:44.597461 17572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:44.597630 17572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:44.598752 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "1052215e95634109b3e246525b43792c"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "e1512ebc2c8c4fd7e6612e71ad072f67"
server_key_iv: "db2abc16fde2301f9b54816b51f0975e"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.599113 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "1052215e95634109b3e246525b43792c"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "e1512ebc2c8c4fd7e6612e71ad072f67"
server_key_iv: "db2abc16fde2301f9b54816b51f0975e"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.600539 17572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:15:44.601276 17587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.601465 17572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:15:44.601532 17572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "1052215e95634109b3e246525b43792c"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "e1512ebc2c8c4fd7e6612e71ad072f67"
server_key_iv: "db2abc16fde2301f9b54816b51f0975e"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.601622 17572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:44.612733 17572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:44.614578 17572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:44.614750 17572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:44.614989 17572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:44.615298 17572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:44.615331 17572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.615355 17572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:44.615401 17572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.621417 17572 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35875
I20260925 10:15:44.621481 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35875 every 8 connection(s)
I20260925 10:15:44.621835 17572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:15:44.626937 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17572
I20260925 10:15:44.627066 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:15:44.627274 3683 external_mini_cluster.cc:1428] Setting key cb7b049606a665fdcc4b045b872d054d
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:44.631798 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.622603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:39129 (local address 127.3.152.254:38901)
0925 10:15:44.622675 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.622676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.622881 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:15:44.623379 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.623380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.623395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.623455 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:44.624484 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.624993 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.625948 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.626136 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.626188 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.628617 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:44.628618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:44.628619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:44.628627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:44.630359 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.630765 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.630767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.630767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.630841 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.631068 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.631069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.631069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.631220 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:44.631292 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:44.631540 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 10:15:44.631649 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:15:44.631881 17703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.622567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:39129)
0925 10:15:44.622745 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.622789 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.623296 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.623530 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.623534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.624183 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:44.624398 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.624404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.625076 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.625077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.625872 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.625873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.626220 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.626221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.626379 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.626889 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:44.626897 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:44.628517 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:44.630444 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.630451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.630453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.630674 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.630903 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.630903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.630904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.631002 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.631285 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:44.631289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:44.631404 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 10:15:44.631486 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 10:15:44.631630 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260925 10:15:44.632366 17701 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:44.632462 17701 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:44.632650 17701 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:44.633183 17510 ts_manager.cc:194] Registered new tserver with Master: 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:44.633813 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:39129
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:15:44.654945 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:44.738116 17706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:44.738337 17706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:44.738363 17706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:44.739663 17706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:44.739704 17706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:44.739733 17706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:15:44.741266 17706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:44.741533 17706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:44.741781 17706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:44.746894 17711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.746912 17714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.746943 17712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.747107 17706 server_base.cc:1034] running on GCE node
I20260925 10:15:44.747390 17706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:44.747632 17706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:44.748797 17706 hybrid_clock.cc:648] HybridClock initialized: now 1790331344748781 us; error 42 us; skew 500 ppm
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:44.751260 17706 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:15:44.751727 17706 webserver.cc:460] Webserver started at http://127.3.152.194:36699/ using document root <none> and password file <none>
I20260925 10:15:44.751924 17706 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:44.752014 17706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:44.752187 17706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:44.753259 17706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "e1e5d583507d4167b687653451ca122f"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "248f63f7e499a22b0da136ba835e63e4"
server_key_iv: "b131201cfac0d3cc63c99e42ff64bfaa"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.753635 17706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "e1e5d583507d4167b687653451ca122f"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "248f63f7e499a22b0da136ba835e63e4"
server_key_iv: "b131201cfac0d3cc63c99e42ff64bfaa"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.754967 17706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:15:44.755719 17721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.755914 17706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:15:44.756038 17706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "e1e5d583507d4167b687653451ca122f"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "248f63f7e499a22b0da136ba835e63e4"
server_key_iv: "b131201cfac0d3cc63c99e42ff64bfaa"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.756098 17706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:44.783774 17706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:44.786307 17706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:44.786425 17706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:44.786615 17706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:44.786957 17706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:44.786993 17706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.787039 17706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:44.787066 17706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.793172 17706 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35997
I20260925 10:15:44.793272 17834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35997 every 8 connection(s)
I20260925 10:15:44.793601 17706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:44.800380 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17706
I20260925 10:15:44.800489 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:15:44.800791 3683 external_mini_cluster.cc:1428] Setting key 0ea549ddceb38801278b1c90a97449ce
I20260925 10:15:44.803504 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.794430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:34197 (local address 127.3.152.254:38901)
0925 10:15:44.794523 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.794525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.794698 (+ 173us) server_negotiation.cc:407] Connection header received
0925 10:15:44.795183 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.795184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.795196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.795262 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:44.796324 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.796835 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.797829 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.798034 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.798096 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.800373 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:44.800375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:44.800375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:44.800383 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:44.802026 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.802498 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.802500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.802500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.802584 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.802835 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.802836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.802837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.802955 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:44.803029 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:44.803327 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 10:15:44.803414 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 10:15:44.803714 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.794405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:34197)
0925 10:15:44.794557 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.794600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.795096 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.795317 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.795321 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.795895 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:44.796220 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.796226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.796925 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.796926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.797746 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.797748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.798117 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.798117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.798230 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.798801 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:44.798810 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:44.800259 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:44.802115 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.802123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.802126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.802396 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.802640 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.802641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.802641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.802765 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.803043 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:44.803050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:44.803189 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 10:15:44.803296 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:15:44.803458 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 10:15:44.804214 17835 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:44.804325 17835 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:44.804518 17835 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:44.804919 17510 ts_manager.cc:194] Registered new tserver with Master: e1e5d583507d4167b687653451ca122f (127.3.152.194:35997)
I20260925 10:15:44.805264 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:34197
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:15:44.827299 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:44.908408 17840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:44.908636 17840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:44.908663 17840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:44.909869 17840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:44.909921 17840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:44.909948 17840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:15:44.911454 17840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:44.911718 17840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:44.912024 17840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:44.917245 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.917254 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:44.917261 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:44.917480 17840 server_base.cc:1034] running on GCE node
I20260925 10:15:44.917734 17840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:44.918020 17840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:44.919207 17840 hybrid_clock.cc:648] HybridClock initialized: now 1790331344919191 us; error 26 us; skew 500 ppm
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:44.921919 17840 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:15:44.922381 17840 webserver.cc:460] Webserver started at http://127.3.152.195:35637/ using document root <none> and password file <none>
I20260925 10:15:44.922618 17840 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:44.922665 17840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:44.922755 17840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:15:44.923789 17840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "5103b74ec819da6d50b40b246e1cd648"
server_key_iv: "065d5464f4ea6fde47b49dcf33e6fe17"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.924165 17840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "5103b74ec819da6d50b40b246e1cd648"
server_key_iv: "065d5464f4ea6fde47b49dcf33e6fe17"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.925468 17840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:44.926265 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.926431 17840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:44.926519 17840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "5103b74ec819da6d50b40b246e1cd648"
server_key_iv: "065d5464f4ea6fde47b49dcf33e6fe17"
server_key_version: "encryptionkey@0"
I20260925 10:15:44.926590 17840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:44.960815 17840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:44.963240 17840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:44.963400 17840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:44.963665 17840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:44.964046 17840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:44.964085 17840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.964109 17840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:44.964123 17840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:44.969842 17840 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:37377
I20260925 10:15:44.969934 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:37377 every 8 connection(s)
I20260925 10:15:44.970237 17840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:15:44.972554 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 17840
I20260925 10:15:44.972628 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:15:44.972807 3683 external_mini_cluster.cc:1428] Setting key 7b299d64e233f0477a9e210e4436fc62
Sep 25 10:15:44 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:44.980139 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.971047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:44735 (local address 127.3.152.254:38901)
0925 10:15:44.971106 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.971136 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.971338 (+ 202us) server_negotiation.cc:407] Connection header received
0925 10:15:44.971935 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.971936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.971950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.972029 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:44.973209 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.973908 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.974934 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.975152 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.975228 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.977392 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:44.977393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:44.977393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:44.977401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:44.978686 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.979101 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.979103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.979103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.979179 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.979438 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.979439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.979439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.979559 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:44.979637 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:44.979925 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 10:15:44.980036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 10:15:44.980319 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.971018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:44735)
0925 10:15:44.971211 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.971249 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.971820 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.972099 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.972103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.972796 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:44.973102 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.973111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.973996 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.973997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.974825 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.974828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.975235 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.975236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:44.975375 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.975841 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:44.975849 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:44.977260 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:44.978776 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.978783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.978786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.979009 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.979248 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:44.979249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:44.979249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:44.979363 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:44.979629 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:44.979635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:44.979781 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:15:44.979888 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 10:15:44.980077 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260925 10:15:44.980801 17969 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:44.980916 17969 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:44.981103 17969 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:44.981541 17510 ts_manager.cc:194] Registered new tserver with Master: e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
I20260925 10:15:44.981937 17510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:44735
I20260925 10:15:44.984745 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:15:44.992197 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.986118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.3.152.254:38901)
0925 10:15:44.986188 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.986190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.986282 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:15:44.986359 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.986360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.986370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.986415 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:44.987015 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.987521 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.988305 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.988508 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.988594 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:44.989822 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:44.989824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:44.989824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:44.989831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:44.990951 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.991303 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.991304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.991305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.991363 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:44.991565 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:44.991566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:44.991566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:44.991731 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:44.991834 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:44.992002 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 10:15:44.992084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:15:44.994037 17510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:15:44.995786 17510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:15:45.002516 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35997 (local address 127.0.0.1:53888)
0925 10:15:44.998559 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.998571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.998627 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.998914 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.998915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.998917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:44.998951 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.998953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.999944 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:44.999946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.001288 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.001291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002133 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.002223 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002237 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.002340 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.002397 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260925 10:15:45.002491 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.3.152.194:35997)
0925 10:15:44.998597 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.998602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.998614 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:15:44.998731 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.998736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.998755 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.998834 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:44.999032 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:44.999842 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.001383 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002041 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002196 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.002362 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.002405 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260925 10:15:45.002754 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.3.152.195:37377)
0925 10:15:44.998694 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.998697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.998733 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:15:44.998775 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.998777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.998794 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.998875 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:44.999062 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.000219 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.001472 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002225 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002301 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002358 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.002602 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.002654 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":86,"threads_started":1}
I20260925 10:15:45.002944 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.0.0.1:37498)
0925 10:15:44.998722 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.998731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.998768 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.999176 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.999177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.999179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:44.999213 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.999214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.000465 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.000466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.001518 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.001520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002404 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.002462 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002466 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.002516 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.002881 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":19,"thread_start_us":101,"threads_started":1}
I20260925 10:15:45.003093 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.3.152.193:35875)
0925 10:15:44.999015 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:44.999020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:44.999029 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:15:44.999067 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:44.999068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:44.999083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:44.999166 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:44.999276 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.000362 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.001604 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002272 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002330 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002376 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.002740 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.003017 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":91,"threads_started":1}
I20260925 10:15:45.003316 17769 tablet_service.cc:1467] Processing CreateTablet for tablet a10a555192954961be185d47f4572cc0 (DEFAULT_TABLE table=test-table [id=3450f3ba775f4e8aa2a149517c876230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.003643 17769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10a555192954961be185d47f4572cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.004182 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:44.998200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37377 (local address 127.0.0.1:53808)
0925 10:15:44.998495 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:44.998542 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:44.998619 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:44.998873 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:44.998876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:44.998878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:44.998951 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:44.998954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.000324 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.000325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.001389 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.001391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.002396 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.002397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.002509 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.002512 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.004055 (+ 1543us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.004108 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":92,"threads_started":1}
I20260925 10:15:45.004693 17635 tablet_service.cc:1467] Processing CreateTablet for tablet a10a555192954961be185d47f4572cc0 (DEFAULT_TABLE table=test-table [id=3450f3ba775f4e8aa2a149517c876230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.005029 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10a555192954961be185d47f4572cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.007125 17986 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Bootstrap starting.
I20260925 10:15:45.007417 17903 tablet_service.cc:1467] Processing CreateTablet for tablet a10a555192954961be185d47f4572cc0 (DEFAULT_TABLE table=test-table [id=3450f3ba775f4e8aa2a149517c876230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.007772 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a10a555192954961be185d47f4572cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.007841 17986 tablet_bootstrap.cc:654] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.008249 17987 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Bootstrap starting.
I20260925 10:15:45.008507 17986 log.cc:824] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.009097 17987 tablet_bootstrap.cc:654] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.009344 17986 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: No bootstrap required, opened a new log
I20260925 10:15:45.009429 17986 ts_tablet_manager.cc:1397] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:15:45.009717 17987 log.cc:824] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.011016 17990 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Bootstrap starting.
I20260925 10:15:45.011291 17986 raft_consensus.cc:348] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.011495 17986 raft_consensus.cc:374] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.011533 17986 raft_consensus.cc:729] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e5d583507d4167b687653451ca122f, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.011665 17990 tablet_bootstrap.cc:654] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.011669 17986 consensus_queue.cc:260] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.012005 17986 ts_tablet_manager.cc:1428] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:15:45.012013 17835 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
I20260925 10:15:45.012256 17990 log.cc:824] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.012712 17987 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: No bootstrap required, opened a new log
I20260925 10:15:45.012799 17987 ts_tablet_manager.cc:1397] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 10:15:45.013306 17990 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: No bootstrap required, opened a new log
I20260925 10:15:45.013374 17990 ts_tablet_manager.cc:1397] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:15:45.014509 17987 raft_consensus.cc:348] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.014668 17987 raft_consensus.cc:374] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.014719 17987 raft_consensus.cc:729] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1052215e95634109b3e246525b43792c, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.014845 17987 consensus_queue.cc:260] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.015169 17987 ts_tablet_manager.cc:1428] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:15:45.015049 17990 raft_consensus.cc:348] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.015193 17701 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
I20260925 10:15:45.015237 17990 raft_consensus.cc:374] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.015285 17990 raft_consensus.cc:729] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58190eeb84f4ffdaa631c3bd4b41eb0, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.015398 17990 consensus_queue.cc:260] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.015718 17990 ts_tablet_manager.cc:1428] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 10:15:45.015743 17969 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
W20260925 10:15:45.044335 17836 tablet.cc:2369] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:15:45.071918 17994 raft_consensus.cc:482] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:45.072115 17994 raft_consensus.cc:504] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.072558 17994 leader_election.cc:288] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:45.076589 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.072813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:55405 (local address 127.3.152.194:35997)
0925 10:15:45.072871 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.072873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.072899 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:15:45.072928 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.072929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.072938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.072992 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:45.073222 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.073987 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.075587 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.076219 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.076289 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.076298 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.076474 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.076515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:15:45.076589 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.072804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:51875 (local address 127.3.152.193:35875)
0925 10:15:45.072869 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.072871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.073032 (+ 161us) server_negotiation.cc:407] Connection header received
0925 10:15:45.073066 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.073068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.073078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.073128 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:45.073313 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.074008 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.075586 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.076210 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.076289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.076298 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.076474 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.076515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:15:45.076608 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.072720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35997 (local address 127.3.152.195:55405)
0925 10:15:45.072781 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.072795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.072840 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.073013 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.073014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.073016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:45.073102 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.073103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.074097 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.074098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.075476 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.075479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.076297 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.076298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.076386 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.076388 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.076449 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.076507 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":48}
I20260925 10:15:45.076720 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.072741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.3.152.195:51875)
0925 10:15:45.072929 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.072938 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.073013 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.073159 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.073159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.073161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:45.073237 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.073238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.074097 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.074098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.075475 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.075479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.076297 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.076298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.076381 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.076386 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.076449 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.076506 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":152,"thread_start_us":115,"threads_started":1}
I20260925 10:15:45.076964 17789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10a555192954961be185d47f4572cc0" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e5d583507d4167b687653451ca122f" is_pre_election: true
I20260925 10:15:45.077134 17789 raft_consensus.cc:2454] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 0.
I20260925 10:15:45.077293 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10a555192954961be185d47f4572cc0" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1052215e95634109b3e246525b43792c" is_pre_election: true
I20260925 10:15:45.077450 17655 raft_consensus.cc:2454] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 0.
I20260925 10:15:45.077406 17859 leader_election.cc:302] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0; no voters:
I20260925 10:15:45.077612 17994 raft_consensus.cc:2781] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:15:45.077689 17994 raft_consensus.cc:482] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:15:45.077718 17994 raft_consensus.cc:3037] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:45.078459 17994 raft_consensus.cc:504] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.078622 17994 leader_election.cc:288] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 election: Requested vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:45.078773 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10a555192954961be185d47f4572cc0" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1052215e95634109b3e246525b43792c"
I20260925 10:15:45.078881 17789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a10a555192954961be185d47f4572cc0" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e5d583507d4167b687653451ca122f"
I20260925 10:15:45.078886 17655 raft_consensus.cc:3037] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:45.078950 17789 raft_consensus.cc:3037] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:45.079843 17789 raft_consensus.cc:2454] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 1.
I20260925 10:15:45.079843 17655 raft_consensus.cc:2454] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 1.
I20260925 10:15:45.080116 17857 leader_election.cc:302] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1052215e95634109b3e246525b43792c, e58190eeb84f4ffdaa631c3bd4b41eb0; no voters:
I20260925 10:15:45.080220 17994 raft_consensus.cc:2781] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:45.080371 17994 raft_consensus.cc:686] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 LEADER]: Becoming Leader. State: Replica: e58190eeb84f4ffdaa631c3bd4b41eb0, State: Running, Role: LEADER
I20260925 10:15:45.080494 17994 consensus_queue.cc:237] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.081398 17509 catalog_manager.cc:5495] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20260925 10:15:45.095711 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.093164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.3.152.195:37377)
0925 10:15:45.093225 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.093228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.093238 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:15:45.093314 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.093314 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.093342 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.093441 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:15:45.093602 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.094081 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.094892 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.095089 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.095169 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.095333 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:15:45.095444 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:15:45.095544 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.095563 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.095583 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:15:45.097592 17655 raft_consensus.cc:1261] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Refusing update from remote peer e58190eeb84f4ffdaa631c3bd4b41eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:45.097590 17789 raft_consensus.cc:1261] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Refusing update from remote peer e58190eeb84f4ffdaa631c3bd4b41eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:45.097988 17996 consensus_queue.cc:1035] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:45.098114 17994 consensus_queue.cc:1035] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:45.100524 17999 mvcc.cc:205] Tried to move back new op lower bound from 7333197189516894208 to 7333197189450526720. Current Snapshot: MvccSnapshot[applied={T|T < 7333197189516894208}]
I20260925 10:15:45.101198 18001 mvcc.cc:205] Tried to move back new op lower bound from 7333197189516894208 to 7333197189450526720. Current Snapshot: MvccSnapshot[applied={T|T < 7333197189516894208}]
I20260925 10:15:45.102980 17509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:45.103084 17509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:45.103899 17509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Sending DeleteTablet for 3 replicas of tablet a10a555192954961be185d47f4572cc0
I20260925 10:15:45.104244 17769 tablet_service.cc:1514] Processing DeleteTablet for tablet a10a555192954961be185d47f4572cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:53888
I20260925 10:15:45.104295 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17572
I20260925 10:15:45.104270 17635 tablet_service.cc:1514] Processing DeleteTablet for tablet a10a555192954961be185d47f4572cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:37498
I20260925 10:15:45.105207 18008 tablet_replica.cc:330] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: stopping tablet replica
I20260925 10:15:45.105340 18008 raft_consensus.cc:2229] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:15:45.105445 18008 raft_consensus.cc:2258] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:45.105466 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet a10a555192954961be185d47f4572cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:53808
I20260925 10:15:45.105849 18008 ts_tablet_manager.cc:1905] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:45.107355 18008 ts_tablet_manager.cc:1918] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:45.107448 18008 log.cc:1196] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/a10a555192954961be185d47f4572cc0
I20260925 10:15:45.107740 18008 ts_tablet_manager.cc:1939] T a10a555192954961be185d47f4572cc0 P e1e5d583507d4167b687653451ca122f: Deleting consensus metadata
I20260925 10:15:45.108273 17497 catalog_manager.cc:4841] TS e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): tablet a10a555192954961be185d47f4572cc0 (table test-table [id=3450f3ba775f4e8aa2a149517c876230]) successfully deleted
W20260925 10:15:45.109678 17495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:35875 (error 108)
I20260925 10:15:45.110253 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.110112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.0.0.1:37504)
0925 10:15:45.110175 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.110218 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:15:45.110328 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17706
W20260925 10:15:45.114121 17495 catalog_manager.cc:4584] TS 1052215e95634109b3e246525b43792c (127.3.152.193:35875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a10a555192954961be185d47f4572cc0: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35875: connect: Connection refused (error 111)
I20260925 10:15:45.114428 18009 tablet_replica.cc:330] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: stopping tablet replica
I20260925 10:15:45.114574 18009 raft_consensus.cc:2229] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:15:45.114727 18009 raft_consensus.cc:2258] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:45.115254 18009 ts_tablet_manager.cc:1905] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:45.116866 18009 ts_tablet_manager.cc:1918] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:45.116957 18009 log.cc:1196] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/a10a555192954961be185d47f4572cc0
I20260925 10:15:45.117229 18009 ts_tablet_manager.cc:1939] T a10a555192954961be185d47f4572cc0 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting consensus metadata
I20260925 10:15:45.117708 17494 catalog_manager.cc:4841] TS e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): tablet a10a555192954961be185d47f4572cc0 (table test-table [id=3450f3ba775f4e8aa2a149517c876230]) successfully deleted
I20260925 10:15:45.118520 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17840
I20260925 10:15:45.123493 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 17478
I20260925 10:15:45.128902 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--webserver_interface=127.3.152.254
--webserver_port=41701
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:45.210675 18011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:45.210893 18011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:45.210916 18011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:45.212049 18011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:45.212083 18011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:45.212097 18011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:45.212110 18011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:45.212122 18011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:45.213542 18011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=41701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:45.213759 18011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:45.213938 18011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:45.223692 18017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.223701 18018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.223694 18020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.223853 18011 server_base.cc:1034] running on GCE node
I20260925 10:15:45.224351 18011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:45.224644 18011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:45.225785 18011 hybrid_clock.cc:648] HybridClock initialized: now 1790331345225775 us; error 30 us; skew 500 ppm
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:45.234575 18011 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:45.235092 18011 webserver.cc:460] Webserver started at http://127.3.152.254:41701/ using document root <none> and password file <none>
I20260925 10:15:45.235368 18011 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:45.235447 18011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:45.237537 18011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:45.238312 18027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.238540 18011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:45.238643 18011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "902a5c75dd4946b58fb6b7c08915180b"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "00185af981683066a415f5593852dd4e"
server_key_iv: "a99ccfc6c53c764fe5a7bbc17837a4f0"
server_key_version: "encryptionkey@0"
I20260925 10:15:45.239078 18011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:45.254005 18011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:45.256935 18011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:45.257097 18011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:45.262398 18011 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:38901
I20260925 10:15:45.262465 18079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:38901 every 8 connection(s)
I20260925 10:15:45.262861 18011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:45.263937 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18011
I20260925 10:15:45.264273 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:35875
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=35709
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:45.265229 18080 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:15:45.266283 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap starting.
I20260925 10:15:45.275281 18080 log.cc:824] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.277612 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:15:45.277964 18080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap complete.
I20260925 10:15:45.280496 18080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:45.280740 18080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.280920 18080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:45.280984 18080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:45.281057 18080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:45.281112 18080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:15:45.282193 18080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:45.282282 18080 leader_election.cc:302] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902a5c75dd4946b58fb6b7c08915180b; no voters:
I20260925 10:15:45.282501 18080 leader_election.cc:288] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:15:45.282541 18084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:15:45.282712 18084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 LEADER]: Becoming Leader. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Running, Role: LEADER
I20260925 10:15:45.282801 18080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:45.282861 18084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:45.283183 18084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902a5c75dd4946b58fb6b7c08915180b. Latest consensus state: current_term: 2 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:45.283188 18085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:45.283309 18084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:45.283342 18085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:45.283913 18089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:45.284957 18089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:45.285071 18089 catalog_manager.cc:1259] Loaded cluster ID: 110cb4d5ba8347708d503c5260e2e0c1
I20260925 10:15:45.285112 18089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:15:45.285403 18089 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 10:15:45.288405 18089 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 10:15:45.290310 18089 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:15:45.290510 18089 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Loaded TSK: 0
I20260925 10:15:45.290730 18089 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:15:45.357681 18082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:45.357937 18082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:45.357981 18082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:45.359266 18082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:45.359332 18082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:45.359378 18082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:15:45.360931 18082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:35875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=35709
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:45.361210 18082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:45.361470 18082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:45.367627 18107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.367695 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.367679 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.367944 18082 server_base.cc:1034] running on GCE node
I20260925 10:15:45.368180 18082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:45.368472 18082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:45.369630 18082 hybrid_clock.cc:648] HybridClock initialized: now 1790331345369618 us; error 28 us; skew 500 ppm
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:45.372520 18082 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:15:45.373174 18082 webserver.cc:460] Webserver started at http://127.3.152.193:35709/ using document root <none> and password file <none>
I20260925 10:15:45.373479 18082 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:45.373544 18082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:45.376394 18082 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 10:15:45.377254 18117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.377455 18082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:45.377576 18082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "1052215e95634109b3e246525b43792c"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "e1512ebc2c8c4fd7e6612e71ad072f67"
server_key_iv: "db2abc16fde2301f9b54816b51f0975e"
server_key_version: "encryptionkey@0"
I20260925 10:15:45.377978 18082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:45.409315 18082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:45.411761 18082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:45.411931 18082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:45.412289 18082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:45.412907 18124 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:15:45.414392 18082 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:15:45.414446 18082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:15:45.414481 18082 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:15:45.415218 18082 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:15:45.415263 18082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:45.415311 18124 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Bootstrap starting.
I20260925 10:15:45.422976 18082 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35875
I20260925 10:15:45.423055 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35875 every 8 connection(s)
I20260925 10:15:45.423673 18082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:15:45.426813 18124 log.cc:824] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.429209 18124 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:15:45.429515 18124 tablet_bootstrap.cc:492] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Bootstrap complete.
I20260925 10:15:45.430307 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18082
I20260925 10:15:45.430610 18124 ts_tablet_manager.cc:1397] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260925 10:15:45.430974 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:35997
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=36699
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:45.431926 18124 raft_consensus.cc:348] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:45.432269 18124 raft_consensus.cc:729] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1052215e95634109b3e246525b43792c, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.432466 18124 consensus_queue.cc:260] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.432777 18124 ts_tablet_manager.cc:1428] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260925 10:15:45.436503 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.424420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:41863)
0925 10:15:45.424687 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.424726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.425318 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.425873 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.425877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.426220 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:45.426490 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.426496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.428144 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.428146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.429267 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.429270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.430028 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.430030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.430183 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.430800 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:45.430809 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:45.432753 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:45.434718 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.434726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.434728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.434965 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.435241 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.435242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.435242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.435404 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.435797 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:45.435805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:45.435946 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.436121 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.436307 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260925 10:15:45.436502 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.424541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:41863 (local address 127.3.152.254:38901)
0925 10:15:45.424780 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.424790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.424835 (+ 45us) server_negotiation.cc:407] Connection header received
0925 10:15:45.425579 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.425587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.425664 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.425852 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:45.426651 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.428002 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.429366 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.429915 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.430027 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.432863 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:45.432867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:45.432867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:45.432879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:45.434609 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.435053 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.435056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.435056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.435158 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.435476 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.435477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.435478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.435718 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:45.435821 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.436179 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.436309 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":120,"threads_started":1}
I20260925 10:15:45.437244 18232 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:45.437352 18232 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:45.437567 18232 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:45.438360 18044 ts_manager.cc:194] Registered new tserver with Master: 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:45.439705 18044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:41863
I20260925 10:15:45.440624 18232 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:15:45.447726 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.439036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.0.0.1:37512)
0925 10:15:45.439247 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.439281 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.439341 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.439619 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.439625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.439953 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:45.440061 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.440064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.441448 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.441449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.442093 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.442095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.442808 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.442809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.442875 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.443442 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:45.443453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:45.444747 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:45.446637 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.446644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.446645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.446850 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.447059 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.447060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.447060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.447115 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.447410 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:45.447416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:45.447459 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.447523 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.447643 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":111,"threads_started":1}
I20260925 10:15:45.447769 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.439089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.3.152.193:35875)
0925 10:15:45.439437 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.439441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.439448 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:15:45.439472 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.439474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.439486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.439557 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:45.440169 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.441358 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.442167 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.442625 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.442699 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.444824 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:45.444831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:45.444832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:45.444838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:45.446555 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.446929 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.446932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.446933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.446994 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.447172 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.447173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.447173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.447349 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:45.447428 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.447573 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.447671 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":65,"threads_started":1}
I20260925 10:15:45.448243 18166 tablet_service.cc:1514] Processing DeleteTablet for tablet a10a555192954961be185d47f4572cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:45 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:37512
I20260925 10:15:45.448792 18241 tablet_replica.cc:330] stopping tablet replica
I20260925 10:15:45.448880 18241 raft_consensus.cc:2229] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:15:45.448936 18241 raft_consensus.cc:2258] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:45.449244 18241 ts_tablet_manager.cc:1905] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:45.451364 18241 ts_tablet_manager.cc:1918] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:45.451428 18241 log.cc:1196] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/a10a555192954961be185d47f4572cc0
I20260925 10:15:45.451660 18241 ts_tablet_manager.cc:1939] T a10a555192954961be185d47f4572cc0 P 1052215e95634109b3e246525b43792c: Deleting consensus metadata
I20260925 10:15:45.452243 18029 catalog_manager.cc:4841] TS 1052215e95634109b3e246525b43792c (127.3.152.193:35875): tablet a10a555192954961be185d47f4572cc0 (table test-table [id=3450f3ba775f4e8aa2a149517c876230]) successfully deleted
W20260925 10:15:45.517691 18237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:45.517937 18237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:45.517975 18237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:45.519135 18237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:45.519202 18237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:45.519246 18237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:15:45.520740 18237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:35997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=36699
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:45.521018 18237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:45.521268 18237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:45.526518 18246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.526546 18249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.526682 18237 server_base.cc:1034] running on GCE node
W20260925 10:15:45.526757 18247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.527024 18237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:45.527302 18237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:45.528473 18237 hybrid_clock.cc:648] HybridClock initialized: now 1790331345528469 us; error 39 us; skew 500 ppm
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:45.530982 18237 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:15:45.531471 18237 webserver.cc:460] Webserver started at http://127.3.152.194:36699/ using document root <none> and password file <none>
I20260925 10:15:45.531706 18237 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:45.531787 18237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:45.533279 18237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:45.533890 18256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.534085 18237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:15:45.534185 18237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "e1e5d583507d4167b687653451ca122f"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "248f63f7e499a22b0da136ba835e63e4"
server_key_iv: "b131201cfac0d3cc63c99e42ff64bfaa"
server_key_version: "encryptionkey@0"
I20260925 10:15:45.534557 18237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:45.545185 18237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:45.547129 18237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:45.547291 18237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:45.547528 18237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:45.547840 18237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:45.547894 18237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.547936 18237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:45.548009 18237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.554972 18237 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35997
I20260925 10:15:45.555066 18369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35997 every 8 connection(s)
I20260925 10:15:45.555461 18237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:15:45.556550 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18237
I20260925 10:15:45.556859 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:37377
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=35637
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:45.567139 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.556313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:39463 (local address 127.3.152.254:38901)
0925 10:15:45.556628 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.556630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.556864 (+ 234us) server_negotiation.cc:407] Connection header received
0925 10:15:45.557544 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.557546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.557559 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.557628 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:45.559047 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.559556 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.561153 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.561355 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.561415 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.563874 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:45.563876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:45.563876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:45.563885 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:45.565709 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.566102 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.566104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.566104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.566173 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.566409 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.566410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.566410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.566575 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:45.566657 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.566904 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.567007 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260925 10:15:45.567157 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.556314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:39463)
0925 10:15:45.556701 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.556760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.557442 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.557764 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.557767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.558644 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:45.558940 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.558947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.559646 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.559647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.561059 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.561063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.561449 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.561450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.561612 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.562161 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:45.562172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:45.563780 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:45.565802 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.565807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.565809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.566026 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.566245 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.566245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.566245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.566341 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.566650 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:45.566656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:45.566770 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.566853 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.566972 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":76,"threads_started":1}
I20260925 10:15:45.567710 18370 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:45.567823 18370 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:45.568044 18370 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:45.568481 18044 ts_manager.cc:194] Registered new tserver with Master: e1e5d583507d4167b687653451ca122f (127.3.152.194:35997)
I20260925 10:15:45.569049 18044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:39463
W20260925 10:15:45.641997 18373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:45.642248 18373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:45.642275 18373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:45.643520 18373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:45.643571 18373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:45.643599 18373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:15:45.645021 18373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=35637
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:45.645231 18373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:45.645416 18373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:45.650712 18378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:45.650676 18379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.650779 18373 server_base.cc:1034] running on GCE node
W20260925 10:15:45.650681 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:45.651081 18373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:45.651291 18373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:45.652431 18373 hybrid_clock.cc:648] HybridClock initialized: now 1790331345652418 us; error 27 us; skew 500 ppm
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:45.655164 18373 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:15:45.655648 18373 webserver.cc:460] Webserver started at http://127.3.152.195:35637/ using document root <none> and password file <none>
I20260925 10:15:45.655880 18373 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:45.655938 18373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:45.657402 18373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:45.658010 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.658206 18373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:15:45.658313 18373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "5103b74ec819da6d50b40b246e1cd648"
server_key_iv: "065d5464f4ea6fde47b49dcf33e6fe17"
server_key_version: "encryptionkey@0"
I20260925 10:15:45.658671 18373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:45.678135 18373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:45.680202 18373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:45.680371 18373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:45.680611 18373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:45.681051 18373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:45.681105 18373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.681157 18373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:45.681186 18373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:45.688050 18373 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:37377
I20260925 10:15:45.688081 18501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:37377 every 8 connection(s)
I20260925 10:15:45.688479 18373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:15:45.692744 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18373
Sep 25 10:15:45 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:15:45.698992 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.689218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39903 (local address 127.3.152.254:38901)
0925 10:15:45.689294 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.689296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.689540 (+ 244us) server_negotiation.cc:407] Connection header received
0925 10:15:45.690012 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.690013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.690024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.690078 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:45.691008 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.691509 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.692583 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.692861 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.692932 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.695597 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:45.695599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:45.695599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:45.695607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:45.697316 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.697801 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.697802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.697802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.697868 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.698198 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.698199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.698199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.698362 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:45.698441 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.698777 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.698868 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:15:45.699198 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.689227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:39903)
0925 10:15:45.689414 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.689450 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.689923 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.690132 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.690136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.690724 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:45.690920 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.690925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.691588 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.691589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.692477 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.692480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.692941 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.692942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.693111 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.693751 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:45.693762 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:45.695488 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:45.697400 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.697409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.697412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.697729 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.697939 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:45.697940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:45.697940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:45.698097 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:45.698440 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:45.698446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:45.698599 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.698727 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.698910 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260925 10:15:45.699764 18502 heartbeater.cc:347] Connected to a master server at 127.3.152.254:38901
I20260925 10:15:45.699877 18502 heartbeater.cc:464] Registering TS with master...
I20260925 10:15:45.700119 18502 heartbeater.cc:511] Master 127.3.152.254:38901 requested a full tablet report, sending...
I20260925 10:15:45.700598 18044 ts_manager.cc:194] Registered new tserver with Master: e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
I20260925 10:15:45.700824 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.693147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.3.152.254:38901)
0925 10:15:45.693266 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.693268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.693329 (+ 61us) server_negotiation.cc:407] Connection header received
0925 10:15:45.693397 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.693397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.693412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.693472 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:45.694268 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.694950 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.695771 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.696085 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.696161 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.697725 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:45.697726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:45.697726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:45.697732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:45.699484 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.699885 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.699887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.699887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.699957 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:45.700203 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:45.700204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:45.700204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:45.700352 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:45.700552 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.700594 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.700724 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260925 10:15:45.701150 18044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:39903
I20260925 10:15:45.701788 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:15:45.704746 18506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.702172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34380 (local address 127.3.152.254:38901)
0925 10:15:45.702236 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.702238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.702345 (+ 107us) server_negotiation.cc:407] Connection header received
0925 10:15:45.702392 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.702392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.702421 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.702481 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:15:45.702692 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.703173 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.703909 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.704203 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.704260 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.704391 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:15:45.704479 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:15:45.704535 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.704564 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.704649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:15:45.705058 18044 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:15:45.706670 18044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:15:45.708941 18166 tablet_service.cc:1467] Processing CreateTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 (DEFAULT_TABLE table=test-table [id=a2d5d440af86454c8ff79c75a46270d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.709164 18166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bcb6ca1e2148828045ea35d1ba3095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.710821 18124 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Bootstrap starting.
I20260925 10:15:45.711154 18124 tablet_bootstrap.cc:654] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.711722 18124 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: No bootstrap required, opened a new log
I20260925 10:15:45.711777 18124 ts_tablet_manager.cc:1397] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:45.712211 18124 raft_consensus.cc:348] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.712301 18124 raft_consensus.cc:374] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.712359 18124 raft_consensus.cc:729] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1052215e95634109b3e246525b43792c, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.712406 18124 consensus_queue.cc:260] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.712481 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.708681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35997 (local address 127.0.0.1:53898)
0925 10:15:45.708724 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.708735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.708773 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.709229 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.709230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.709231 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:45.709280 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.709282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.710310 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.710311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.711284 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.711287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.712191 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.712192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.712268 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.712294 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.712353 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.712406 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":1}
I20260925 10:15:45.712507 18124 ts_tablet_manager.cc:1428] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 10:15:45.712787 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.708827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.3.152.194:35997)
0925 10:15:45.709036 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.709042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.709052 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:15:45.709092 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.709094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.709111 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.709200 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:45.709365 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.710203 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.711382 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.712122 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.712191 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.712265 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.712655 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.712707 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260925 10:15:45.713306 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.708682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37377 (local address 127.0.0.1:53838)
0925 10:15:45.709213 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:45.709224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:45.709272 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:45.709513 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:45.709514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:45.709516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:45.709552 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.709553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.711005 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.711006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.712081 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:45.712084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.713077 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.713078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:45.713138 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.713142 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:15:45.713199 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 10:15:45.713239 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":292,"thread_start_us":89,"threads_started":1}
I20260925 10:15:45.713328 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:45.708796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.3.152.195:37377)
0925 10:15:45.709349 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:45.709353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:45.709363 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:15:45.709401 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:45.709403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:45.709418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:45.709502 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:45.709614 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.710900 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.712181 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:45.712914 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:45.712982 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:45.713040 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:45.713233 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 10:15:45.713269 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":129,"threads_started":1}
I20260925 10:15:45.713533 18299 tablet_service.cc:1467] Processing CreateTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 (DEFAULT_TABLE table=test-table [id=a2d5d440af86454c8ff79c75a46270d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.713871 18299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bcb6ca1e2148828045ea35d1ba3095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.714066 18436 tablet_service.cc:1467] Processing CreateTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 (DEFAULT_TABLE table=test-table [id=a2d5d440af86454c8ff79c75a46270d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:15:45.714401 18436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91bcb6ca1e2148828045ea35d1ba3095. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:15:45.716741 18512 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Bootstrap starting.
I20260925 10:15:45.717344 18513 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Bootstrap starting.
I20260925 10:15:45.717502 18512 tablet_bootstrap.cc:654] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.717947 18513 tablet_bootstrap.cc:654] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Neither blocks nor log segments found. Creating new log.
I20260925 10:15:45.718014 18512 log.cc:824] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.718427 18513 log.cc:824] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:45.718854 18512 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: No bootstrap required, opened a new log
I20260925 10:15:45.718928 18512 ts_tablet_manager.cc:1397] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:15:45.719225 18513 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: No bootstrap required, opened a new log
I20260925 10:15:45.719306 18513 ts_tablet_manager.cc:1397] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:15:45.720588 18512 raft_consensus.cc:348] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.720741 18512 raft_consensus.cc:374] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.720764 18512 raft_consensus.cc:729] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e5d583507d4167b687653451ca122f, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.720911 18512 consensus_queue.cc:260] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.720897 18513 raft_consensus.cc:348] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.721079 18513 raft_consensus.cc:374] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:15:45.721135 18513 raft_consensus.cc:729] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58190eeb84f4ffdaa631c3bd4b41eb0, State: Initialized, Role: FOLLOWER
I20260925 10:15:45.721196 18512 ts_tablet_manager.cc:1428] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:15:45.721264 18370 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
I20260925 10:15:45.721278 18513 consensus_queue.cc:260] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:45.721544 18513 ts_tablet_manager.cc:1428] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:15:45.721623 18502 heartbeater.cc:503] Master 127.3.152.254:38901 was elected leader, sending a full tablet report...
W20260925 10:15:45.806084 18371 tablet.cc:2369] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:15:45.924535 18233 tablet.cc:2369] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:15:45.939054 18503 tablet.cc:2369] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:15:46.117764 18517 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:46.117909 18517 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:46.118373 18517 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:46.121577 18508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.118838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:37693 (local address 127.3.152.194:35997)
0925 10:15:46.118897 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:46.118900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:46.118933 (+ 33us) server_negotiation.cc:407] Connection header received
0925 10:15:46.119036 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:46.119037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:46.119049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:46.119139 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:46.119291 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.119772 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.120849 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.121214 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.121302 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:46.121311 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:46.121464 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:15:46.121502 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 10:15:46.121614 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.118811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:58191 (local address 127.3.152.193:35875)
0925 10:15:46.118925 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:46.118929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:46.118946 (+ 17us) server_negotiation.cc:407] Connection header received
0925 10:15:46.118974 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:46.118975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:46.118998 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:46.119112 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:15:46.119305 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.119800 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.120852 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.121277 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.121369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:46.121452 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:46.121484 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 10:15:46.121523 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260925 10:15:46.121613 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.118637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.3.152.195:58191)
0925 10:15:46.118704 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.118718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:46.118785 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:46.119131 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:46.119132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:46.119134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:46.119213 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:46.119215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.119891 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.119892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:46.120750 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:46.120753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.121367 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.121368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:46.121434 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:46.121437 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:15:46.121478 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 10:15:46.121517 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":52}
I20260925 10:15:46.121629 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.118698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35997 (local address 127.3.152.195:37693)
0925 10:15:46.118880 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.118890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:46.118944 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:46.119138 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:46.119139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:46.119141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:15:46.119219 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:46.119221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.119870 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.119872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:46.120750 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:46.120752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.121301 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.121302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:46.121372 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:46.121376 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:15:46.121444 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 10:15:46.121489 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260925 10:15:46.121958 18186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bcb6ca1e2148828045ea35d1ba3095" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1052215e95634109b3e246525b43792c" is_pre_election: true
I20260925 10:15:46.122144 18186 raft_consensus.cc:2454] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 0.
I20260925 10:15:46.122213 18324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bcb6ca1e2148828045ea35d1ba3095" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e5d583507d4167b687653451ca122f" is_pre_election: true
I20260925 10:15:46.122377 18390 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1052215e95634109b3e246525b43792c, e58190eeb84f4ffdaa631c3bd4b41eb0; no voters:
I20260925 10:15:46.122416 18324 raft_consensus.cc:2454] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 0.
I20260925 10:15:46.122517 18517 raft_consensus.cc:2781] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:15:46.122576 18517 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:15:46.122604 18517 raft_consensus.cc:3037] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:46.123342 18517 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:46.123497 18517 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 election: Requested vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), 1052215e95634109b3e246525b43792c (127.3.152.193:35875)
I20260925 10:15:46.123672 18324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bcb6ca1e2148828045ea35d1ba3095" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1e5d583507d4167b687653451ca122f"
I20260925 10:15:46.123736 18186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91bcb6ca1e2148828045ea35d1ba3095" candidate_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1052215e95634109b3e246525b43792c"
I20260925 10:15:46.123782 18324 raft_consensus.cc:3037] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:46.123812 18186 raft_consensus.cc:3037] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:15:46.124683 18186 raft_consensus.cc:2454] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 1.
I20260925 10:15:46.124683 18324 raft_consensus.cc:2454] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e58190eeb84f4ffdaa631c3bd4b41eb0 in term 1.
I20260925 10:15:46.124915 18392 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0; no voters:
I20260925 10:15:46.125005 18517 raft_consensus.cc:2781] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:15:46.125159 18517 raft_consensus.cc:686] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 LEADER]: Becoming Leader. State: Replica: e58190eeb84f4ffdaa631c3bd4b41eb0, State: Running, Role: LEADER
I20260925 10:15:46.125268 18517 consensus_queue.cc:237] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:46.126096 18044 catalog_manager.cc:5495] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } health_report { overall_health: UNKNOWN } } }
I20260925 10:15:46.153587 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.151299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.3.152.195:37377)
0925 10:15:46.151352 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:46.151354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:46.151362 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:15:46.151446 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:46.151446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:46.151466 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:46.151520 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:15:46.151684 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.152168 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.152900 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:46.153083 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:46.153137 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:46.153255 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:15:46.153356 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:15:46.153414 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:46.153486 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 10:15:46.153503 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:15:46.155562 18186 raft_consensus.cc:1261] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Refusing update from remote peer e58190eeb84f4ffdaa631c3bd4b41eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:46.155562 18324 raft_consensus.cc:1261] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Refusing update from remote peer e58190eeb84f4ffdaa631c3bd4b41eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:15:46.155931 18520 consensus_queue.cc:1035] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:46.156119 18517 consensus_queue.cc:1035] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:15:46.159386 18524 mvcc.cc:205] Tried to move back new op lower bound from 7333197193850073088 to 7333197193729589248. Current Snapshot: MvccSnapshot[applied={T|T < 7333197193850073088}]
I20260925 10:15:46.160820 18043 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:46.160943 18043 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:15:46.161821 18043 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Sending DeleteTablet for 3 replicas of tablet 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:46.162187 18436 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:46 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:53838
I20260925 10:15:46.162279 18299 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:46 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:53898
I20260925 10:15:46.162412 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18082
I20260925 10:15:46.162447 18166 tablet_service.cc:1514] Processing DeleteTablet for tablet 91bcb6ca1e2148828045ea35d1ba3095 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:15:46 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:37512
I20260925 10:15:46.162544 18533 tablet_replica.cc:330] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: stopping tablet replica
I20260925 10:15:46.162636 18533 raft_consensus.cc:2229] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:15:46.162738 18533 raft_consensus.cc:2258] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:46.163404 18533 ts_tablet_manager.cc:1905] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:46.165126 18532 tablet_replica.cc:330] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: stopping tablet replica
I20260925 10:15:46.165280 18532 raft_consensus.cc:2229] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:15:46.165421 18532 raft_consensus.cc:2258] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:15:46.165501 18533 ts_tablet_manager.cc:1918] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:46.165611 18533 log.cc:1196] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:46.165942 18533 ts_tablet_manager.cc:1939] T 91bcb6ca1e2148828045ea35d1ba3095 P e1e5d583507d4167b687653451ca122f: Deleting consensus metadata
I20260925 10:15:46.166052 18532 ts_tablet_manager.cc:1905] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:15:46.166437 18031 catalog_manager.cc:4841] TS e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): tablet 91bcb6ca1e2148828045ea35d1ba3095 (table test-table [id=a2d5d440af86454c8ff79c75a46270d3]) successfully deleted
I20260925 10:15:46.167511 18532 ts_tablet_manager.cc:1918] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:15:46.167596 18532 log.cc:1196] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:46.167862 18532 ts_tablet_manager.cc:1939] T 91bcb6ca1e2148828045ea35d1ba3095 P e58190eeb84f4ffdaa631c3bd4b41eb0: Deleting consensus metadata
I20260925 10:15:46.168318 18028 catalog_manager.cc:4841] TS e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): tablet 91bcb6ca1e2148828045ea35d1ba3095 (table test-table [id=a2d5d440af86454c8ff79c75a46270d3]) successfully deleted
W20260925 10:15:46.169538 18029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:35875 (error 108)
I20260925 10:15:46.169895 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18237
I20260925 10:15:46.170058 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.169914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35875 (local address 127.0.0.1:37520)
0925 10:15:46.169973 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.170018 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 10:15:46.171339 18029 catalog_manager.cc:4584] TS 1052215e95634109b3e246525b43792c (127.3.152.193:35875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91bcb6ca1e2148828045ea35d1ba3095: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35875: connect: Connection refused (error 111)
I20260925 10:15:46.175381 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18373
I20260925 10:15:46.180156 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18011
I20260925 10:15:46.185930 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--webserver_interface=127.3.152.254
--webserver_port=41701
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:15:46.271107 18535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:46.271516 18535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:46.271625 18535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:46.272907 18535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:15:46.272953 18535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:46.272966 18535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:46.272979 18535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:15:46.272991 18535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:15:46.274459 18535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=41701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:46.274668 18535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:46.274835 18535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:46.279534 18541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.279547 18540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.279597 18543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:46.279803 18535 server_base.cc:1034] running on GCE node
I20260925 10:15:46.280154 18535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:46.280467 18535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:46.281687 18535 hybrid_clock.cc:648] HybridClock initialized: now 1790331346281666 us; error 48 us; skew 500 ppm
Sep 25 10:15:46 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:46.284232 18535 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:15:46.284693 18535 webserver.cc:460] Webserver started at http://127.3.152.254:41701/ using document root <none> and password file <none>
I20260925 10:15:46.284948 18535 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:46.285019 18535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:46.286480 18535 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.287292 18550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.287508 18535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.287722 18535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "902a5c75dd4946b58fb6b7c08915180b"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "00185af981683066a415f5593852dd4e"
server_key_iv: "a99ccfc6c53c764fe5a7bbc17837a4f0"
server_key_version: "encryptionkey@0"
I20260925 10:15:46.288201 18535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:46.304581 18535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:46.306800 18535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:46.306972 18535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:46.311038 18535 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:38901
I20260925 10:15:46.311153 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:38901 every 8 connection(s)
I20260925 10:15:46.311506 18535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:15:46.313288 18603 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:15:46.313973 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap starting.
I20260925 10:15:46.321341 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18535
I20260925 10:15:46.321673 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:35875
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=35709
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:46.322686 18603 log.cc:824] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:46.325088 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:15:46.325419 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b: Bootstrap complete.
I20260925 10:15:46.327757 18603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:46.328073 18603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Initialized, Role: FOLLOWER
I20260925 10:15:46.328267 18603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:46.328351 18603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:15:46.328405 18603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:15:46.328454 18603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:15:46.329499 18603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:46.329631 18603 leader_election.cc:302] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 902a5c75dd4946b58fb6b7c08915180b; no voters:
I20260925 10:15:46.329838 18603 leader_election.cc:288] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:15:46.329875 18607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:15:46.329982 18607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [term 3 LEADER]: Becoming Leader. State: Replica: 902a5c75dd4946b58fb6b7c08915180b, State: Running, Role: LEADER
I20260925 10:15:46.330124 18603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:15:46.330158 18607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } }
I20260925 10:15:46.330533 18608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:46.330547 18609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 902a5c75dd4946b58fb6b7c08915180b. Latest consensus state: current_term: 3 leader_uuid: "902a5c75dd4946b58fb6b7c08915180b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "902a5c75dd4946b58fb6b7c08915180b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 38901 } } }
I20260925 10:15:46.330652 18608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:46.330657 18609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902a5c75dd4946b58fb6b7c08915180b [sys.catalog]: This master's current role is: LEADER
I20260925 10:15:46.330885 18614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:15:46.331776 18614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:15:46.331929 18614 catalog_manager.cc:1259] Loaded cluster ID: 110cb4d5ba8347708d503c5260e2e0c1
I20260925 10:15:46.332201 18614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 10:15:46.332445 18614 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa59c5b99ad google::LogMessage::Fail() at ??:0
@ 0x7fa59c5bc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa59c5b94de google::LogMessage::Flush() at ??:0
@ 0x7fa59c5baa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa5a44da0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa5a44e38b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa59e4870d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa59e47c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa59f2e06db start_thread at ??:0
@ 0x7fa59b0b761f clone at ??:0
W20260925 10:15:46.420539 18605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:46.420771 18605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:46.420797 18605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:46.422029 18605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:46.422079 18605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:46.422109 18605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:15:46.423617 18605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:35875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=35709
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:46.423913 18605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:46.424201 18605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:46.429631 18639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.429709 18640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.429852 18642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:46.429759 18605 server_base.cc:1034] running on GCE node
I20260925 10:15:46.430191 18605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:46.430510 18605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:46.431711 18605 hybrid_clock.cc:648] HybridClock initialized: now 1790331346431690 us; error 39 us; skew 500 ppm
Sep 25 10:15:46 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:46.434571 18605 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:15:46.435179 18605 webserver.cc:460] Webserver started at http://127.3.152.193:35709/ using document root <none> and password file <none>
I20260925 10:15:46.435420 18605 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:46.435488 18605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:46.436973 18605 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.437747 18649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.437963 18605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.438058 18605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "1052215e95634109b3e246525b43792c"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "e1512ebc2c8c4fd7e6612e71ad072f67"
server_key_iv: "db2abc16fde2301f9b54816b51f0975e"
server_key_version: "encryptionkey@0"
I20260925 10:15:46.438516 18605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:46.447830 18605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:46.450277 18605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:46.450449 18605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:46.450690 18605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:46.451174 18656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:15:46.453199 18605 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:15:46.453263 18605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:15:46.453330 18605 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:15:46.453982 18605 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:15:46.454038 18605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:15:46.454124 18656 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Bootstrap starting.
I20260925 10:15:46.461241 18605 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35875
I20260925 10:15:46.461640 18605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:15:46.464227 18763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35875 every 8 connection(s)
I20260925 10:15:46.465179 18656 log.cc:824] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Log is configured to *not* fsync() on all Append() calls
I20260925 10:15:46.467217 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18605
I20260925 10:15:46.467504 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:35997
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=36699
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:46.467900 18656 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:15:46.468259 18656 tablet_bootstrap.cc:492] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Bootstrap complete.
I20260925 10:15:46.469494 18656 ts_tablet_manager.cc:1397] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20260925 10:15:46.470964 18656 raft_consensus.cc:348] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:46.471324 18656 raft_consensus.cc:729] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1052215e95634109b3e246525b43792c, State: Initialized, Role: FOLLOWER
I20260925 10:15:46.471505 18656 consensus_queue.cc:260] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:46.473007 18656 ts_tablet_manager.cc:1428] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
W20260925 10:15:46.530614 18766 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:15:46.462311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:48785)
0925 10:15:46.462484 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.462530 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:46.463004 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:46.530389 (+ 67385us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260925 10:15:46.530908 18764 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:38901 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:38901: Client connection negotiation failed: client connection to 127.3.152.254:38901: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:15:46.531337 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.531202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:38447)
0925 10:15:46.531275 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.531317 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:15:46.531810 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.531713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:57435)
0925 10:15:46.531766 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.531791 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:15:46.531956 18764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:15:46.557410 18768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:46.557641 18768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:46.557668 18768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:46.558859 18768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:46.558907 18768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:46.558936 18768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:15:46.560495 18768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:35997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=36699
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:46.560757 18768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:46.561000 18768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:15:46.566258 18768 server_base.cc:1034] running on GCE node
W20260925 10:15:46.566344 18777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.566255 18775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.566255 18774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:46.566668 18768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:46.566926 18768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:46.568097 18768 hybrid_clock.cc:648] HybridClock initialized: now 1790331346568087 us; error 38 us; skew 500 ppm
Sep 25 10:15:46 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:46.570822 18768 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:15:46.571218 18768 webserver.cc:460] Webserver started at http://127.3.152.194:36699/ using document root <none> and password file <none>
I20260925 10:15:46.571398 18768 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:46.571439 18768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:46.572811 18768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.573385 18784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.573557 18768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:15:46.573673 18768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "e1e5d583507d4167b687653451ca122f"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "248f63f7e499a22b0da136ba835e63e4"
server_key_iv: "b131201cfac0d3cc63c99e42ff64bfaa"
server_key_version: "encryptionkey@0"
I20260925 10:15:46.574002 18768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:46.594550 18768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:46.596824 18768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:46.596946 18768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:46.597193 18768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:46.597530 18768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:46.597620 18768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.597645 18768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:46.597661 18768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.603361 18768 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35997
I20260925 10:15:46.603439 18897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35997 every 8 connection(s)
I20260925 10:15:46.603765 18768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:15:46.604990 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.604521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:60967)
0925 10:15:46.604748 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.604833 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
W20260925 10:15:46.605458 18898 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:38901 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:38901: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
I20260925 10:15:46.605521 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18768
I20260925 10:15:46.605851 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.605713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:55493)
0925 10:15:46.605770 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.605820 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 10:15:46.605841 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:37377
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:38901
--webserver_port=35637
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:15:46.606140 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.606052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:41917)
0925 10:15:46.606097 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.606127 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:15:46.606235 18898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:15:46.689303 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:15:46.689523 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:15:46.689551 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:15:46.690745 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:15:46.690791 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:15:46.690819 18901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:15:46.692305 18901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:37377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=35637
--tserver_master_addrs=127.3.152.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:15:46.692559 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:15:46.692898 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:15:46.697947 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.697969 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:15:46.697953 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:15:46.698328 18901 server_base.cc:1034] running on GCE node
I20260925 10:15:46.698542 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:15:46.698880 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:15:46.700031 18901 hybrid_clock.cc:648] HybridClock initialized: now 1790331346699999 us; error 43 us; skew 500 ppm
Sep 25 10:15:46 dist-test-slave-dzwq krb5kdc[17465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:15:46.702718 18901 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:15:46.703202 18901 webserver.cc:460] Webserver started at http://127.3.152.195:35637/ using document root <none> and password file <none>
I20260925 10:15:46.703389 18901 fs_manager.cc:359] Metadata directory not provided
I20260925 10:15:46.703430 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:15:46.704865 18901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.705492 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.705736 18901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:15:46.705856 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0"
format_stamp: "Formatted at 2026-09-25 10:15:44 on dist-test-slave-dzwq"
server_key: "5103b74ec819da6d50b40b246e1cd648"
server_key_iv: "065d5464f4ea6fde47b49dcf33e6fe17"
server_key_version: "encryptionkey@0"
I20260925 10:15:46.706211 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:15:46.718588 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:15:46.720786 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:15:46.720963 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:15:46.721210 18901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:15:46.721539 18901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:15:46.721603 18901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.721680 18901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:15:46.721712 18901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:15:46.727669 18901 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:37377
I20260925 10:15:46.727720 19029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:37377 every 8 connection(s)
I20260925 10:15:46.728091 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:15:46.729166 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.728771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:53551)
0925 10:15:46.728926 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.729001 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
W20260925 10:15:46.729586 19030 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:38901 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:38901: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
I20260925 10:15:46.729887 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.729790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:54789)
0925 10:15:46.729842 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.729869 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 10:15:46.730190 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:46.730139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:42601)
0925 10:15:46.730161 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:46.730179 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 10:15:46.730319 19030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:15:46.730986 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 18901
I20260925 10:15:47.532655 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:47.532409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:49511)
0925 10:15:47.532544 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:47.532588 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260925 10:15:47.606807 19035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:47.606540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:50265)
0925 10:15:47.606719 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:47.606758 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":108,"threads_started":1}
I20260925 10:15:47.730927 19036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:47.730698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:47901)
0925 10:15:47.730829 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:47.730873 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260925 10:15:48.228048 19037 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:48.228200 19037 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:48.228564 19037 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
Sep 25 10:15:48 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.194@KRBTEST.COM
Sep 25 10:15:48 dist-test-slave-dzwq krb5kdc[17465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331346, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.195@KRBTEST.COM
I20260925 10:15:48.241676 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.228858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:46281 (local address 127.3.152.194:35997)
0925 10:15:48.229030 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:48.229043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:48.229072 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:15:48.229673 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:48.229677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:48.229743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:48.229902 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:48.230978 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.232423 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234046 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.234704 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234849 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:48.237849 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:48.237854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:48.237855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:48.237869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:48.239794 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240346 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:48.240350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:48.240350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:48.240459 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240732 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:48.240734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:48.240734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:48.241055 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:48.241152 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:48.241420 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 10:15:48.241544 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260925 10:15:48.241714 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.228751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:37377 (local address 127.3.152.193:36197)
0925 10:15:48.228885 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:48.228896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:48.228956 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:48.229976 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:48.229979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:48.230521 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:48.230843 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:48.230852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.232542 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.232543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:48.233945 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:48.233948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.234866 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:48.235009 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:48.235693 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:48.235704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:48.238097 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:48.239950 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:48.239951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:48.239952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:48.240254 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:48.240535 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:48.240537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:48.240661 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:48.241102 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:48.241110 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:48.241250 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 10:15:48.241355 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:15:48.241586 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":123,"thread_start_us":74,"threads_started":1}
I20260925 10:15:48.241758 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.228881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:36197 (local address 127.3.152.195:37377)
0925 10:15:48.229030 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:15:48.229043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 10:15:48.229073 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:15:48.229674 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:15:48.229677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:15:48.229743 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:15:48.229902 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:15:48.230977 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.232423 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234046 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.234704 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234848 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:48.238211 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:15:48.238216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:15:48.238216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:15:48.238229 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:15:48.239869 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240360 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:48.240370 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:48.240371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:48.240458 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240741 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:15:48.240741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:15:48.240742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:15:48.241035 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:15:48.241099 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:15:48.241469 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 10:15:48.241629 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":45,"threads_started":1}
I20260925 10:15:48.241794 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.228704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35997 (local address 127.3.152.193:46281)
0925 10:15:48.228885 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:48.228896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:15:48.228956 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:15:48.229976 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:15:48.229979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:15:48.230521 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:15:48.230843 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:48.230852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.232541 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.232542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:48.233945 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:15:48.233948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:15:48.234866 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:15:48.234867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:15:48.235010 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:15:48.235705 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:15:48.235710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:15:48.237702 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:15:48.239888 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:48.239896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:48.239899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:48.240254 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:48.240536 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:15:48.240537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:15:48.240537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:15:48.240662 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:15:48.241145 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:15:48.241146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:15:48.241251 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 10:15:48.241357 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 10:15:48.241642 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":196,"thread_start_us":70,"threads_started":1}
W20260925 10:15:48.242588 18650 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
W20260925 10:15:48.243165 18653 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:48.243249 18653 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1052215e95634109b3e246525b43792c; no voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0
I20260925 10:15:48.243363 19037 raft_consensus.cc:2726] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:15:48.533427 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.533237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:42817)
0925 10:15:48.533322 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:48.533368 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:15:48.607439 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.607228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:47339)
0925 10:15:48.607354 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:48.607397 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260925 10:15:48.731768 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:48.731524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:55203)
0925 10:15:48.731654 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:48.731696 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260925 10:15:49.534219 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:49.533954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:44255)
0925 10:15:49.534109 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:49.534160 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260925 10:15:49.608126 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:49.607901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:51071)
0925 10:15:49.608041 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:49.608083 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260925 10:15:49.732421 19046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:49.732214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:52351)
0925 10:15:49.732342 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:49.732378 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260925 10:15:49.844602 19047 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:49.844707 19047 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:49.844878 19047 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
W20260925 10:15:49.845353 18653 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
W20260925 10:15:49.845494 18650 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:49.845592 18650 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1052215e95634109b3e246525b43792c; no voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0
I20260925 10:15:49.845739 19047 raft_consensus.cc:2726] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:15:50.534968 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:50.534735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:50535)
0925 10:15:50.534865 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:50.534914 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260925 10:15:50.608835 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:50.608612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:54805)
0925 10:15:50.608745 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:50.608780 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260925 10:15:50.733184 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:50.732936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:54687)
0925 10:15:50.733076 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:50.733122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260925 10:15:51.535784 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:51.535535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:52335)
0925 10:15:51.535682 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:51.535734 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
I20260925 10:15:51.609644 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:51.609391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:37579)
0925 10:15:51.609555 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:51.609598 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260925 10:15:51.734042 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:51.733802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:53571)
0925 10:15:51.733953 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:51.733996 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260925 10:15:51.852937 19054 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:51.853051 19054 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:51.853319 19054 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
W20260925 10:15:51.853775 18653 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
W20260925 10:15:51.853863 18650 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:51.853947 18650 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1052215e95634109b3e246525b43792c; no voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0
I20260925 10:15:51.854053 19054 raft_consensus.cc:2726] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:15:52.536592 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:52.536337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:51427)
0925 10:15:52.536492 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:52.536535 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260925 10:15:52.610356 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:52.610135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:59879)
0925 10:15:52.610270 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:52.610308 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260925 10:15:52.734928 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:52.734592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:49313)
0925 10:15:52.734813 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:52.734868 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":130,"threads_started":1}
I20260925 10:15:53.537396 19058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:53.537148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:49887)
0925 10:15:53.537303 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:53.537347 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260925 10:15:53.611047 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:53.610817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:47583)
0925 10:15:53.610960 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:53.611002 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260925 10:15:53.735813 19060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:53.735529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:45837)
0925 10:15:53.735681 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:53.735753 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260925 10:15:54.538252 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:54.538018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:56627)
0925 10:15:54.538146 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:54.538183 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 10:15:54.611755 19062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:54.611535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:55617)
0925 10:15:54.611668 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:54.611708 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260925 10:15:54.736539 19063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:54.736319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:45873)
0925 10:15:54.736445 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:54.736491 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260925 10:15:55.539090 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:55.538836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:55687)
0925 10:15:55.538987 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:55.539038 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260925 10:15:55.612572 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:55.612319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:50599)
0925 10:15:55.612475 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:55.612523 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260925 10:15:55.713320 19066 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:15:55.713428 19066 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:15:55.713666 19066 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
W20260925 10:15:55.714088 18650 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
W20260925 10:15:55.714221 18653 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:15:55.714257 18653 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1052215e95634109b3e246525b43792c; no voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0
I20260925 10:15:55.714345 19066 raft_consensus.cc:2726] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:15:55.737426 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:55.737146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:60599)
0925 10:15:55.737329 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:55.737378 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260925 10:15:56.539927 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:56.539711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:58725)
0925 10:15:56.539836 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:56.539879 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260925 10:15:56.613329 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:56.613111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:52651)
0925 10:15:56.613252 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:56.613284 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260925 10:15:56.738324 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:56.738028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:41407)
0925 10:15:56.738226 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:56.738274 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260925 10:15:57.540757 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:57.540514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:59201)
0925 10:15:57.540668 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:57.540710 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260925 10:15:57.614112 19072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:57.613864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:50275)
0925 10:15:57.614012 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:57.614063 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260925 10:15:57.739053 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:57.738829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:58191)
0925 10:15:57.738966 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:57.739004 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260925 10:15:58.541548 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:58.541258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:34643)
0925 10:15:58.541420 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:58.541463 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260925 10:15:58.614915 19075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:58.614649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:44089)
0925 10:15:58.614823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:58.614867 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260925 10:15:58.739867 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:58.739618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:56645)
0925 10:15:58.739766 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:58.739812 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260925 10:15:59.542392 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:59.542166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:52023)
0925 10:15:59.542296 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:59.542340 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260925 10:15:59.615628 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:59.615368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:46535)
0925 10:15:59.615522 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:59.615575 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 10:15:59.740617 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:15:59.740399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:39179)
0925 10:15:59.740526 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 10:15:59.740566 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260925 10:16:00.543061 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:00.542830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:56277)
0925 10:16:00.542973 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:00.543011 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260925 10:16:00.616461 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:00.616248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:39679)
0925 10:16:00.616381 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:00.616414 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260925 10:16:00.741359 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:00.741128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:60515)
0925 10:16:00.741266 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:00.741311 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260925 10:16:01.544078 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:01.543753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.193:46479)
0925 10:16:01.543909 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:01.543957 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260925 10:16:01.617421 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:01.617111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.194:54221)
0925 10:16:01.617321 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:01.617373 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":96,"threads_started":1}
I20260925 10:16:01.648505 19085 raft_consensus.cc:482] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:01.648650 19085 raft_consensus.cc:504] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e5d583507d4167b687653451ca122f" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35997 } } peers { permanent_uuid: "e58190eeb84f4ffdaa631c3bd4b41eb0" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 37377 } } peers { permanent_uuid: "1052215e95634109b3e246525b43792c" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35875 } }
I20260925 10:16:01.648880 19085 leader_election.cc:288] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e5d583507d4167b687653451ca122f (127.3.152.194:35997), e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377)
W20260925 10:16:01.649286 18653 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1e5d583507d4167b687653451ca122f (127.3.152.194:35997): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
W20260925 10:16:01.649390 18650 leader_election.cc:341] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e58190eeb84f4ffdaa631c3bd4b41eb0 (127.3.152.195:37377): Not found: Tablet not found: 91bcb6ca1e2148828045ea35d1ba3095
I20260925 10:16:01.649477 18650 leader_election.cc:302] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1052215e95634109b3e246525b43792c; no voters: e1e5d583507d4167b687653451ca122f, e58190eeb84f4ffdaa631c3bd4b41eb0
I20260925 10:16:01.649566 19085 raft_consensus.cc:2726] T 91bcb6ca1e2148828045ea35d1ba3095 P 1052215e95634109b3e246525b43792c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:16:01.736532 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18605
I20260925 10:16:01.741338 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18768
I20260925 10:16:01.743068 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:01.742841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:38901 (local address 127.3.152.195:48599)
0925 10:16:01.742983 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:01.743021 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:38901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260925 10:16:01.745745 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 18901
2026-09-25T10:16:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18627 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:16:01 dist-test-slave-dzwq krb5kdc[19089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:16:01 dist-test-slave-dzwq krb5kdc[19089](info): set up 2 sockets
Sep 25 10:16:01 dist-test-slave-dzwq krb5kdc[19089](info): commencing operation
krb5kdc: starting...
W20260925 10:16:02.792773 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:16:02 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:16:02.914896 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:34129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:02.997936 19102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:02.998193 19102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:02.998235 19102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:02.999410 19102 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:16:02.999477 19102 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:16:02.999500 19102 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:16:02.999518 19102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:16:02.999545 19102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:02.999577 19102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:02.999603 19102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:16:02.999630 19102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:16:03.001165 19102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:34129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:03.001475 19102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:03.001722 19102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:03.006933 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.007021 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.006934 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:03.007237 19102 server_base.cc:1034] running on GCE node
I20260925 10:16:03.007475 19102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:03.007812 19102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:03.008978 19102 hybrid_clock.cc:648] HybridClock initialized: now 1790331363008963 us; error 25 us; skew 500 ppm
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:03.011889 19102 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:16:03.012367 19102 webserver.cc:460] Webserver started at http://127.3.152.254:42379/ using document root <none> and password file <none>
I20260925 10:16:03.012611 19102 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:03.012658 19102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:03.012830 19102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:16:03.013935 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "d799418173904693954d9e1728608ae8"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "03b4b9f3a2bef5cfbd70e578e80ac16e"
server_key_iv: "50b58a9a0e0c95d1324cf6c54cbbcb9d"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.014317 19102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "d799418173904693954d9e1728608ae8"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "03b4b9f3a2bef5cfbd70e578e80ac16e"
server_key_iv: "50b58a9a0e0c95d1324cf6c54cbbcb9d"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.015770 19102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:16:03.016571 19117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.016839 19102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:16:03.016947 19102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "d799418173904693954d9e1728608ae8"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "03b4b9f3a2bef5cfbd70e578e80ac16e"
server_key_iv: "50b58a9a0e0c95d1324cf6c54cbbcb9d"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.017024 19102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:03.038558 19102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:03.041189 19102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:03.041451 19102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:03.045871 19102 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:34129
I20260925 10:16:03.045946 19169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:34129 every 8 connection(s)
I20260925 10:16:03.046303 19102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:16:03.047335 19170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:03.049825 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap starting.
I20260925 10:16:03.050122 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19102
I20260925 10:16:03.050242 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:16:03.050451 3683 external_mini_cluster.cc:1428] Setting key 299e93d98894dfe5975acf52c220eb44
I20260925 10:16:03.050595 19170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:03.050966 19170 log.cc:824] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:03.051712 19170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: No bootstrap required, opened a new log
I20260925 10:16:03.054176 19170 raft_consensus.cc:348] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:03.054364 19170 raft_consensus.cc:374] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:03.054400 19170 raft_consensus.cc:729] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d799418173904693954d9e1728608ae8, State: Initialized, Role: FOLLOWER
I20260925 10:16:03.054527 19170 consensus_queue.cc:260] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:03.054600 19170 raft_consensus.cc:388] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:16:03.054629 19170 raft_consensus.cc:482] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:16:03.054667 19170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:03.055333 19170 raft_consensus.cc:504] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:03.055471 19170 leader_election.cc:302] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d799418173904693954d9e1728608ae8; no voters:
I20260925 10:16:03.055677 19170 leader_election.cc:288] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:16:03.055702 19175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:16:03.055826 19175 raft_consensus.cc:686] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 LEADER]: Becoming Leader. State: Replica: d799418173904693954d9e1728608ae8, State: Running, Role: LEADER
I20260925 10:16:03.055922 19175 consensus_queue.cc:237] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:03.056002 19170 sys_catalog.cc:564] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:16:03.056332 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d799418173904693954d9e1728608ae8. Latest consensus state: current_term: 1 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:03.056402 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:16:03.057202 19176 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:03.057303 19176 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
W20260925 10:16:03.058943 19191 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:16:03.059273 19191 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:16:03.059346 19179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:16:03.059504 19179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:16:03.061465 19179 catalog_manager.cc:1347] Generated new cluster ID: c9272cffeb8f4cdeb59c3e641fa8f58a
I20260925 10:16:03.061539 19179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:16:03.062994 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.051041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.3.152.254:34129)
0925 10:16:03.051245 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.051254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.051291 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:16:03.051743 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.051749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.051824 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.052025 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:03.052881 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.053871 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.054956 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.055391 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.055555 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.058529 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:03.058534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:03.058535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:03.058549 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:03.060847 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.061325 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.061328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.061329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.061465 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.061708 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.061709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.061710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.062000 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:03.062134 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.062648 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.062785 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260925 10:16:03.066283 19179 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:16:03.066996 19179 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:16:03.071198 19179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Generated new TSK 0
I20260925 10:16:03.071379 19179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:16:03.103447 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:03.186617 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:03.186897 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:03.186947 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:03.188212 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:03.188295 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:03.188344 19196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:16:03.189929 19196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:03.190186 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:03.190457 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:03.196693 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.196640 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:03.196861 19196 server_base.cc:1034] running on GCE node
W20260925 10:16:03.196930 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:03.197220 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:03.197506 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:03.198688 19196 hybrid_clock.cc:648] HybridClock initialized: now 1790331363198637 us; error 70 us; skew 500 ppm
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:03.201398 19196 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:16:03.201887 19196 webserver.cc:460] Webserver started at http://127.3.152.193:42683/ using document root <none> and password file <none>
I20260925 10:16:03.202140 19196 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:03.202224 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:03.202376 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:16:03.203557 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "f8771dd2da4d46f0b0243e02340d7920"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "084385f93205ebc12e4dfe9538541260"
server_key_iv: "46595d0a7254bf78d83d0dcec63c6c4c"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.203938 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "f8771dd2da4d46f0b0243e02340d7920"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "084385f93205ebc12e4dfe9538541260"
server_key_iv: "46595d0a7254bf78d83d0dcec63c6c4c"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.205389 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:03.206270 19211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.206513 19196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:03.206619 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "f8771dd2da4d46f0b0243e02340d7920"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "084385f93205ebc12e4dfe9538541260"
server_key_iv: "46595d0a7254bf78d83d0dcec63c6c4c"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.206705 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:03.225519 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:03.227490 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:03.227674 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:03.227923 19196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:03.228291 19196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:03.228355 19196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.228392 19196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:03.228434 19196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.234370 19196 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34553
I20260925 10:16:03.234453 19324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34553 every 8 connection(s)
I20260925 10:16:03.234810 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:16:03.239229 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19196
I20260925 10:16:03.239372 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:16:03.239560 3683 external_mini_cluster.cc:1428] Setting key 2269afd3182fc1eb0467d4bf127e384a
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:03.245445 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.235558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:58509)
0925 10:16:03.235716 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.235757 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.236457 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.236746 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.236751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.237388 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:03.237600 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.237606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.238371 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.238372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.239221 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.239223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.239730 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.239731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.239870 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.240499 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:03.240510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:03.241965 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:03.243724 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.243733 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.243736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.244134 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.244396 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.244397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.244397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.244538 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.244892 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:03.244898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:03.245030 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.245131 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.245269 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260925 10:16:03.245463 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.235573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:58509 (local address 127.3.152.254:34129)
0925 10:16:03.235625 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.235627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.235854 (+ 227us) server_negotiation.cc:407] Connection header received
0925 10:16:03.236585 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.236586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.236600 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.236665 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:03.237730 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.238260 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.239330 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.239610 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.239702 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.242076 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:03.242085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:03.242085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:03.242094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:03.243626 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.244242 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.244244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.244244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.244312 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.244636 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.244637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.244638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.244818 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:03.244897 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.245206 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.245321 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:16:03.246090 19325 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:03.246212 19325 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:03.246440 19325 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:03.246981 19134 ts_manager.cc:194] Registered new tserver with Master: f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193:34553)
I20260925 10:16:03.247622 19134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:58509
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:16:03.267288 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:03.352530 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:03.352752 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:03.352778 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:03.353947 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:03.353986 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:03.354012 19330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:16:03.355403 19330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:03.355671 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:03.355896 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:03.361279 19335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.361328 19336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.361307 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:03.361511 19330 server_base.cc:1034] running on GCE node
I20260925 10:16:03.361776 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:03.362030 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:03.363175 19330 hybrid_clock.cc:648] HybridClock initialized: now 1790331363363166 us; error 29 us; skew 500 ppm
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:03.365931 19330 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:16:03.366456 19330 webserver.cc:460] Webserver started at http://127.3.152.194:40927/ using document root <none> and password file <none>
I20260925 10:16:03.366693 19330 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:03.366766 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:03.366925 19330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:16:03.368067 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "476e5a272ae34d7fa9881512075365d2"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "de5d02f5c1d6e42fe0c8e673c6ecfd3d"
server_key_iv: "94f46c838bb9496746a69b2f37c6e02a"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.368446 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "476e5a272ae34d7fa9881512075365d2"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "de5d02f5c1d6e42fe0c8e673c6ecfd3d"
server_key_iv: "94f46c838bb9496746a69b2f37c6e02a"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.369884 19330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:16:03.370671 19345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.370883 19330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:03.370991 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "476e5a272ae34d7fa9881512075365d2"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "de5d02f5c1d6e42fe0c8e673c6ecfd3d"
server_key_iv: "94f46c838bb9496746a69b2f37c6e02a"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.371068 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:03.389827 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:03.392412 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:03.392588 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:03.392882 19330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:03.393226 19330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:03.393281 19330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.393317 19330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:03.393358 19330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.399520 19330 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:46645
I20260925 10:16:03.399564 19458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:46645 every 8 connection(s)
I20260925 10:16:03.399909 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:16:03.402645 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19330
I20260925 10:16:03.402763 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:16:03.402988 3683 external_mini_cluster.cc:1428] Setting key f47728dfebfcce05cae2cc59ecc6d717
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:03.411803 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.400791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:50223 (local address 127.3.152.254:34129)
0925 10:16:03.400862 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.400864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.401045 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:16:03.401521 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.401521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.401531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.401593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:03.402658 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.403499 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.404750 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.405111 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.405194 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.408219 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:03.408221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:03.408221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:03.408232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:03.410184 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.410666 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.410668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.410668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.410744 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.411011 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.411012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.411012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.411196 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:03.411276 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.411559 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.411670 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:16:03.411916 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.400759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:50223)
0925 10:16:03.400911 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.400953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.401441 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.401654 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.401657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.402263 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:03.402468 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.402473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.403631 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.403632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.404655 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.404658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.405231 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.405232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.405402 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.406012 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:03.406023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:03.408097 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:03.410319 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.410327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.410329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.410578 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.410822 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.410822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.410823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.410932 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.411272 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:03.411277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:03.411408 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.411499 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.411629 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260925 10:16:03.412458 19459 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:03.412595 19459 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:03.412802 19459 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:03.413239 19134 ts_manager.cc:194] Registered new tserver with Master: 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
I20260925 10:16:03.413846 19134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:50223
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:16:03.433140 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:03.516388 19464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:03.516654 19464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:03.516696 19464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:03.517935 19464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:03.518009 19464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:03.518056 19464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:16:03.519593 19464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:03.519840 19464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:03.520057 19464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:03.525424 19472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.525416 19469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:03.525416 19470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:03.525753 19464 server_base.cc:1034] running on GCE node
I20260925 10:16:03.525974 19464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:03.526221 19464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:03.527382 19464 hybrid_clock.cc:648] HybridClock initialized: now 1790331363527358 us; error 34 us; skew 500 ppm
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:03.529876 19464 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:16:03.530336 19464 webserver.cc:460] Webserver started at http://127.3.152.195:44403/ using document root <none> and password file <none>
I20260925 10:16:03.530570 19464 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:03.530640 19464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:03.530795 19464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:16:03.531899 19464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "440606ab2d1ab44b9687beec7ad2f657"
server_key_iv: "d47d65f6be9a8ef0a1504bca2e2dcca0"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.532334 19464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "440606ab2d1ab44b9687beec7ad2f657"
server_key_iv: "d47d65f6be9a8ef0a1504bca2e2dcca0"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.533789 19464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:03.534617 19479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.534795 19464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:03.534902 19464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "440606ab2d1ab44b9687beec7ad2f657"
server_key_iv: "d47d65f6be9a8ef0a1504bca2e2dcca0"
server_key_version: "encryptionkey@0"
I20260925 10:16:03.534976 19464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:03.544457 19464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:03.546564 19464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:03.546741 19464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:03.546981 19464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:03.547310 19464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:03.547364 19464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.547405 19464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:03.547432 19464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:03.554461 19464 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:38129
I20260925 10:16:03.554535 19592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:38129 every 8 connection(s)
I20260925 10:16:03.554977 19464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:16:03.558470 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19464
I20260925 10:16:03.558576 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:16:03.558779 3683 external_mini_cluster.cc:1428] Setting key 6e2c2c8107309e61bcad94c650f8dc7d
Sep 25 10:16:03 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331363, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:03.565654 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.556481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:59633 (local address 127.3.152.254:34129)
0925 10:16:03.556552 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.556555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.556567 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:16:03.556821 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.556823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.556835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.556912 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:03.558247 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.559064 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.560446 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.560827 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.560911 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.563094 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:03.563096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:03.563096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:03.563102 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:03.564445 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.564863 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.564864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.564864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.564919 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.565130 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.565131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.565131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.565236 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:03.565286 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.565525 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.565587 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:16:03.565903 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.555772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:59633)
0925 10:16:03.555934 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.556017 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.556676 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.557018 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.557022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.557811 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:03.558125 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.558134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.559170 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.559171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.560330 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.560333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.560922 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.560923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.561061 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.561557 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:03.561565 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:03.563015 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:03.564532 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.564537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.564539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.564795 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.564977 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:03.564978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:03.564978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:03.565076 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:03.565291 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:03.565296 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:03.565412 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.565496 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.565619 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260925 10:16:03.566345 19593 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:03.566479 19593 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:03.566723 19593 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:03.567137 19134 ts_manager.cc:194] Registered new tserver with Master: e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129)
I20260925 10:16:03.567523 19134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:59633
I20260925 10:16:03.570327 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:16:03.578512 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.571794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.3.152.254:34129)
0925 10:16:03.571878 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.571879 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.571997 (+ 118us) server_negotiation.cc:407] Connection header received
0925 10:16:03.572078 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.572079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.572090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.572145 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:03.572712 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.573201 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.574022 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.574235 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.574319 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.575685 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:03.575686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:03.575686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:03.575694 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:03.577232 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.577573 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.577574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.577574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.577634 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:03.577858 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:03.577859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:03.577860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:03.578028 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:03.578127 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.578259 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.578332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 10:16:03.580235 19134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:16:03.581979 19134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:16:03.588981 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34553 (local address 127.0.0.1:51064)
0925 10:16:03.584660 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.584704 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.584773 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.585166 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.585169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.585172 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:03.585262 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.585264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586634 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.586635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.587848 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.587850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588734 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.588810 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588813 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.588863 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.588895 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":12,"thread_start_us":44,"threads_started":1}
I20260925 10:16:03.589007 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.3.152.193:34553)
0925 10:16:03.584992 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.584997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.585007 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:16:03.585050 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.585052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.585069 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.585151 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:03.585347 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586529 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.587923 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588614 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588731 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588786 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.588875 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.588920 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260925 10:16:03.589154 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.3.152.194:46645)
0925 10:16:03.585312 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.585315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.585325 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:16:03.585361 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.585363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.585378 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.585448 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:03.585656 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586734 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.587828 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588578 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588654 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588717 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.589022 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.589064 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":75,"threads_started":1}
I20260925 10:16:03.589222 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.0.0.1:35462)
0925 10:16:03.584930 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.584941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.584986 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.585525 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.585526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.585528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:03.585590 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.585591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586835 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.586836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.587742 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.587744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588753 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.588829 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588832 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.588882 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.588923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260925 10:16:03.589838 19259 tablet_service.cc:1467] Processing CreateTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 (DEFAULT_TABLE table=test-table [id=496acb589ff548389fa1146be976efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:03.589980 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.3.152.195:38129)
0925 10:16:03.584956 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.584961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.585047 (+ 86us) server_negotiation.cc:407] Connection header received
0925 10:16:03.585180 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.585183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.585211 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.585291 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:03.585568 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586367 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588012 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588509 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588684 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.588968 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.589021 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260925 10:16:03.590107 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.584522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38129 (local address 127.0.0.1:43122)
0925 10:16:03.584961 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.584974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.585042 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.585407 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.585408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.585410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:03.585467 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.585468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.586473 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.586474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.587849 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.587851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.588627 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.588628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.588709 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.588723 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.588793 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.588857 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":13,"thread_start_us":150,"threads_started":1}
I20260925 10:16:03.590224 19259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d974ae86dc5447d9ab09fe08d72b0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:03.590116 19393 tablet_service.cc:1467] Processing CreateTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 (DEFAULT_TABLE table=test-table [id=496acb589ff548389fa1146be976efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:03.590458 19393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d974ae86dc5447d9ab09fe08d72b0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:03.590783 19527 tablet_service.cc:1467] Processing CreateTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 (DEFAULT_TABLE table=test-table [id=496acb589ff548389fa1146be976efe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:03.591105 19527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d974ae86dc5447d9ab09fe08d72b0543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:03.593359 19610 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap starting.
I20260925 10:16:03.593906 19611 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Bootstrap starting.
I20260925 10:16:03.594168 19610 tablet_bootstrap.cc:654] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:03.594628 19611 tablet_bootstrap.cc:654] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:03.594831 19610 log.cc:824] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:03.595175 19611 log.cc:824] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:03.595497 19612 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Bootstrap starting.
I20260925 10:16:03.595695 19610 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: No bootstrap required, opened a new log
I20260925 10:16:03.595790 19610 ts_tablet_manager.cc:1397] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:03.595906 19611 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: No bootstrap required, opened a new log
I20260925 10:16:03.596139 19611 ts_tablet_manager.cc:1397] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:03.596177 19612 tablet_bootstrap.cc:654] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:03.596743 19612 log.cc:824] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:03.597533 19612 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: No bootstrap required, opened a new log
I20260925 10:16:03.597615 19612 ts_tablet_manager.cc:1397] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:03.597725 19610 raft_consensus.cc:348] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.597903 19610 raft_consensus.cc:374] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:03.597946 19610 raft_consensus.cc:729] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Initialized, Role: FOLLOWER
I20260925 10:16:03.598062 19610 consensus_queue.cc:260] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.598346 19610 ts_tablet_manager.cc:1428] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:16:03.598248 19611 raft_consensus.cc:348] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.598410 19611 raft_consensus.cc:374] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:03.598462 19611 raft_consensus.cc:729] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476e5a272ae34d7fa9881512075365d2, State: Initialized, Role: FOLLOWER
I20260925 10:16:03.598497 19325 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
I20260925 10:16:03.598584 19611 consensus_queue.cc:260] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.598830 19611 ts_tablet_manager.cc:1428] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:16:03.598989 19459 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
I20260925 10:16:03.599502 19612 raft_consensus.cc:348] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.599635 19612 raft_consensus.cc:374] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:03.599673 19612 raft_consensus.cc:729] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0d330e9fa4a0bbbf34c4e2e2b5c36, State: Initialized, Role: FOLLOWER
I20260925 10:16:03.599749 19612 consensus_queue.cc:260] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.599936 19612 ts_tablet_manager.cc:1428] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:16:03.600028 19593 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
I20260925 10:16:03.647190 19616 raft_consensus.cc:482] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:03.647323 19616 raft_consensus.cc:504] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.647782 19616 leader_election.cc:288] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645), e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129)
W20260925 10:16:03.650532 19460 tablet.cc:2369] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:16:03.650980 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.648063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:34087 (local address 127.3.152.195:38129)
0925 10:16:03.648123 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.648126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.648232 (+ 106us) server_negotiation.cc:407] Connection header received
0925 10:16:03.648270 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.648271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.648295 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.648369 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:03.648616 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.649091 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650110 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.650517 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650647 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.650657 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.650873 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.650920 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:16:03.650980 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.648202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:43781 (local address 127.3.152.194:46645)
0925 10:16:03.648250 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.648252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.648350 (+ 98us) server_negotiation.cc:407] Connection header received
0925 10:16:03.648422 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.648423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.648433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.648507 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:03.648662 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.649119 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650079 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.650532 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650610 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.650619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.650873 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.650912 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:16:03.651010 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.648045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38129 (local address 127.3.152.193:34087)
0925 10:16:03.648097 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.648111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.648182 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.648388 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.648389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.648391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:03.648474 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.648476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.649198 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.649199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.649977 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.649979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.650696 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.650783 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.650785 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.650849 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.650916 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":41}
I20260925 10:16:03.651017 19619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.648059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.3.152.193:43781)
0925 10:16:03.648268 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.648278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:03.648364 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:03.648508 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:03.648509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:03.648511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:03.648583 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.648585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.649219 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.649220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.649979 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:03.649981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.650704 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.650705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:03.650781 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.650785 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:16:03.650850 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:16:03.650911 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260925 10:16:03.651437 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d974ae86dc5447d9ab09fe08d72b0543" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476e5a272ae34d7fa9881512075365d2" is_pre_election: true
I20260925 10:16:03.651437 19547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d974ae86dc5447d9ab09fe08d72b0543" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" is_pre_election: true
I20260925 10:16:03.651589 19413 raft_consensus.cc:2454] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 0.
I20260925 10:16:03.651590 19547 raft_consensus.cc:2454] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 0.
I20260925 10:16:03.651880 19215 leader_election.cc:302] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 476e5a272ae34d7fa9881512075365d2, e2c0d330e9fa4a0bbbf34c4e2e2b5c36, f8771dd2da4d46f0b0243e02340d7920; no voters:
I20260925 10:16:03.652078 19616 raft_consensus.cc:2781] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:16:03.652122 19616 raft_consensus.cc:482] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:16:03.652137 19616 raft_consensus.cc:3037] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:03.652827 19616 raft_consensus.cc:504] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.652972 19616 leader_election.cc:288] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 election: Requested vote from peers 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645), e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129)
I20260925 10:16:03.653132 19547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d974ae86dc5447d9ab09fe08d72b0543" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
I20260925 10:16:03.653239 19547 raft_consensus.cc:3037] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:03.653280 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d974ae86dc5447d9ab09fe08d72b0543" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476e5a272ae34d7fa9881512075365d2"
I20260925 10:16:03.653407 19413 raft_consensus.cc:3037] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:03.654006 19547 raft_consensus.cc:2454] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 1.
I20260925 10:16:03.654248 19212 leader_election.cc:302] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c0d330e9fa4a0bbbf34c4e2e2b5c36, f8771dd2da4d46f0b0243e02340d7920; no voters:
I20260925 10:16:03.654404 19616 raft_consensus.cc:2781] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:16:03.654422 19413 raft_consensus.cc:2454] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 1.
I20260925 10:16:03.654615 19616 raft_consensus.cc:686] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 LEADER]: Becoming Leader. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Running, Role: LEADER
I20260925 10:16:03.654760 19616 consensus_queue.cc:237] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:03.655701 19134 catalog_manager.cc:5495] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "f8771dd2da4d46f0b0243e02340d7920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } health_report { overall_health: UNKNOWN } } }
I20260925 10:16:03.665078 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.662320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.3.152.193:34553)
0925 10:16:03.662368 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:03.662371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:03.662384 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:16:03.662427 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:03.662428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:03.662450 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:03.662542 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:16:03.662688 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.663229 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.664166 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:03.664387 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:03.664482 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:03.664674 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:16:03.664796 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:16:03.664904 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:03.664932 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 10:16:03.664959 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:16:03.667093 19413 raft_consensus.cc:1261] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Refusing update from remote peer f8771dd2da4d46f0b0243e02340d7920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:16:03.667104 19547 raft_consensus.cc:1261] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Refusing update from remote peer f8771dd2da4d46f0b0243e02340d7920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:16:03.667539 19620 consensus_queue.cc:1035] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:16:03.667662 19616 consensus_queue.cc:1035] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:16:03.670457 19623 mvcc.cc:205] Tried to move back new op lower bound from 7333197265577181184 to 7333197265530380288. Current Snapshot: MvccSnapshot[applied={T|T < 7333197265577181184}]
I20260925 10:16:03.670584 19624 mvcc.cc:205] Tried to move back new op lower bound from 7333197265577181184 to 7333197265530380288. Current Snapshot: MvccSnapshot[applied={T|T < 7333197265577181184}]
I20260925 10:16:03.671072 19625 mvcc.cc:205] Tried to move back new op lower bound from 7333197265577181184 to 7333197265530380288. Current Snapshot: MvccSnapshot[applied={T|T < 7333197265577181184}]
I20260925 10:16:03.672823 19133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:16:03.672935 19133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:16:03.673839 19133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Sending DeleteTablet for 3 replicas of tablet d974ae86dc5447d9ab09fe08d72b0543
I20260925 10:16:03.674252 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19196
I20260925 10:16:03.674325 19393 tablet_service.cc:1514] Processing DeleteTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:03 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:35462
I20260925 10:16:03.674862 19527 tablet_service.cc:1514] Processing DeleteTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:03 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:43122
I20260925 10:16:03.675140 19634 tablet_replica.cc:330] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: stopping tablet replica
I20260925 10:16:03.675271 19634 raft_consensus.cc:2229] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:03.675408 19634 raft_consensus.cc:2258] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:03.675916 19634 ts_tablet_manager.cc:1905] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:03.677873 19634 ts_tablet_manager.cc:1918] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:16:03.677979 19634 log.cc:1196] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/d974ae86dc5447d9ab09fe08d72b0543
I20260925 10:16:03.678340 19634 ts_tablet_manager.cc:1939] T d974ae86dc5447d9ab09fe08d72b0543 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting consensus metadata
I20260925 10:16:03.678807 19118 catalog_manager.cc:4841] TS e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): tablet d974ae86dc5447d9ab09fe08d72b0543 (table test-table [id=496acb589ff548389fa1146be976efe8]) successfully deleted
I20260925 10:16:03.680267 19633 tablet_replica.cc:330] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: stopping tablet replica
W20260925 10:16:03.680294 19119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:34553 (error 108)
I20260925 10:16:03.680413 19633 raft_consensus.cc:2229] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:03.680543 19633 raft_consensus.cc:2258] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:03.680583 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19330
I20260925 10:16:03.682638 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.682468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34553 (local address 127.0.0.1:51078)
0925 10:16:03.682549 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.682604 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 10:16:03.682781 19119 catalog_manager.cc:4584] TS f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193:34553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d974ae86dc5447d9ab09fe08d72b0543: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34553: connect: Connection refused (error 111)
I20260925 10:16:03.685530 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19464
I20260925 10:16:03.685587 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:03.685442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.0.0.1:35474)
0925 10:16:03.685502 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:03.685541 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.194:46645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:16:03.690779 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19102
I20260925 10:16:23.696345 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--webserver_interface=127.3.152.254
--webserver_port=42379
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:34129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:23.779462 19636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:23.779695 19636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:23.779718 19636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:23.780936 19636 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:16:23.780980 19636 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:16:23.780993 19636 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:16:23.781004 19636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:16:23.781018 19636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:23.781060 19636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:23.781090 19636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:16:23.781108 19636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:16:23.782604 19636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:34129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:23.782859 19636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:23.783107 19636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:23.791958 19642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:23.791944 19643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:23.792173 19636 server_base.cc:1034] running on GCE node
W20260925 10:16:23.791944 19645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:23.792585 19636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:23.792905 19636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:23.794073 19636 hybrid_clock.cc:648] HybridClock initialized: now 1790331383794056 us; error 43 us; skew 500 ppm
Sep 25 10:16:23 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:23.797887 19636 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:16:23.798506 19636 webserver.cc:460] Webserver started at http://127.3.152.254:42379/ using document root <none> and password file <none>
I20260925 10:16:23.798803 19636 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:23.798897 19636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:23.800803 19636 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:16:23.801666 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:23.801851 19636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:23.801956 19636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "d799418173904693954d9e1728608ae8"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "03b4b9f3a2bef5cfbd70e578e80ac16e"
server_key_iv: "50b58a9a0e0c95d1324cf6c54cbbcb9d"
server_key_version: "encryptionkey@0"
I20260925 10:16:23.802500 19636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:23.824853 19636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:23.826973 19636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:23.827112 19636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:23.831373 19636 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:34129
I20260925 10:16:23.831431 19704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:34129 every 8 connection(s)
I20260925 10:16:23.831861 19636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:16:23.833499 19705 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:16:23.834282 19705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap starting.
I20260925 10:16:23.841923 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19636
I20260925 10:16:23.842248 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:34553
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=42683
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:16:23.843216 19705 log.cc:824] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:23.845326 19705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:16:23.845731 19705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap complete.
I20260925 10:16:23.848242 19705 raft_consensus.cc:348] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:23.848515 19705 raft_consensus.cc:729] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d799418173904693954d9e1728608ae8, State: Initialized, Role: FOLLOWER
I20260925 10:16:23.848739 19705 consensus_queue.cc:260] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:23.848825 19705 raft_consensus.cc:388] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:16:23.848886 19705 raft_consensus.cc:482] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:16:23.848937 19705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 10:16:23.850011 19705 raft_consensus.cc:504] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:23.850142 19705 leader_election.cc:302] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d799418173904693954d9e1728608ae8; no voters:
I20260925 10:16:23.850349 19705 leader_election.cc:288] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 10:16:23.850394 19709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 10:16:23.850510 19709 raft_consensus.cc:686] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 LEADER]: Becoming Leader. State: Replica: d799418173904693954d9e1728608ae8, State: Running, Role: LEADER
I20260925 10:16:23.850610 19709 consensus_queue.cc:237] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:23.850646 19705 sys_catalog.cc:564] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:16:23.850883 19709 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d799418173904693954d9e1728608ae8. Latest consensus state: current_term: 2 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:23.850962 19709 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:16:23.850888 19710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:23.851302 19710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:16:23.851738 19713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:16:23.852638 19713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:16:23.852743 19713 catalog_manager.cc:1259] Loaded cluster ID: c9272cffeb8f4cdeb59c3e641fa8f58a
I20260925 10:16:23.852769 19713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:16:23.853997 19713 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:16:23.854208 19713 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Loaded TSK: 0
I20260925 10:16:23.864645 19713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Generated new TSK 1
I20260925 10:16:23.864758 19713 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 10:16:23.928088 19707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:23.928316 19707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:23.928360 19707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:23.929587 19707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:23.929654 19707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:23.929701 19707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:16:23.931397 19707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:34553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=42683
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:23.931818 19707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:23.932142 19707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:23.938323 19735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:23.938306 19733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:23.938496 19707 server_base.cc:1034] running on GCE node
W20260925 10:16:23.938308 19732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:23.938886 19707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:23.939191 19707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:23.940356 19707 hybrid_clock.cc:648] HybridClock initialized: now 1790331383940333 us; error 36 us; skew 500 ppm
Sep 25 10:16:23 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:23.948274 19707 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:16:23.948875 19707 webserver.cc:460] Webserver started at http://127.3.152.193:42683/ using document root <none> and password file <none>
I20260925 10:16:23.949147 19707 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:23.949249 19707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:23.952112 19707 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:23.952980 19742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:23.953226 19707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:23.953365 19707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "f8771dd2da4d46f0b0243e02340d7920"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "084385f93205ebc12e4dfe9538541260"
server_key_iv: "46595d0a7254bf78d83d0dcec63c6c4c"
server_key_version: "encryptionkey@0"
I20260925 10:16:23.953773 19707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:23.983700 19707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:23.986261 19707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:23.986441 19707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:23.986701 19707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:23.987185 19749 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:16:23.989095 19707 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:16:23.989184 19707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:16:23.989265 19707 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:16:23.990010 19707 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:16:23.990072 19707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:16:23.990185 19749 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap starting.
I20260925 10:16:23.997138 19707 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34553
I20260925 10:16:23.997272 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34553 every 8 connection(s)
I20260925 10:16:23.997620 19707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:16:23.998202 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19707
I20260925 10:16:23.998528 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:46645
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=40927
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:16:24.002017 19749 log.cc:824] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:24.004478 19749 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:16:24.004823 19749 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap complete.
I20260925 10:16:24.006002 19749 ts_tablet_manager.cc:1397] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260925 10:16:24.007072 19749 raft_consensus.cc:348] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:24.007326 19749 raft_consensus.cc:729] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Initialized, Role: FOLLOWER
I20260925 10:16:24.007485 19749 consensus_queue.cc:260] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:24.007829 19749 ts_tablet_manager.cc:1428] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:24.011869 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:23.998495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:47327 (local address 127.3.152.254:34129)
0925 10:16:24.000132 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.000141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.000188 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:16:24.000370 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.000380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.000459 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.000628 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.001362 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.002697 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.004372 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.005031 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.005141 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.008264 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.008268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.008268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.008279 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.009961 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.010513 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.010515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.010516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.010607 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.010911 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.010911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.010912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.011148 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.011225 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.011608 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.011721 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":125,"threads_started":1}
I20260925 10:16:24.012032 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:23.998382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:47327)
0925 10:16:23.998576 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:23.998620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:23.999414 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.000690 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.000696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.000996 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:24.001195 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.001200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.002839 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.002840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.004269 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.004273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.005160 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.005161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.005357 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.006018 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:24.006029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:24.008138 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:24.010051 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.010060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.010062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.010425 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.010678 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.010679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.010679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.010833 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.011220 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:24.011227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:24.011426 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.011560 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.011782 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260925 10:16:24.012840 19857 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:24.012961 19857 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:24.013161 19857 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:24.013973 19669 ts_manager.cc:194] Registered new tserver with Master: f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193:34553)
I20260925 10:16:24.015126 19669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:47327
I20260925 10:16:24.016148 19857 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.193@KRBTEST.COM
I20260925 10:16:24.023664 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.014561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34553 (local address 127.0.0.1:48860)
0925 10:16:24.014777 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.014809 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.014868 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.015752 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.015759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.016139 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:24.016286 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.016289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.017307 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.017308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.017901 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.017903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.018406 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.018406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.018494 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.019021 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:24.019025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:24.020553 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:24.022547 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.022555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.022556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.022760 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.022981 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.022982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.022982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.023035 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.023372 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:24.023378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:24.023423 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.023486 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.023588 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260925 10:16:24.023732 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.014637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.3.152.193:34553)
0925 10:16:24.015518 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.015543 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.015555 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:16:24.015590 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.015592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.015607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.015679 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.016384 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.017188 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.017987 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.018303 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.018401 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.020648 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.020658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.020658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.020666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.022460 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.022842 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.022845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.022845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.022915 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.023108 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.023109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.023109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.023306 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.023387 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.023542 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.023642 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":87,"threads_started":1}
I20260925 10:16:24.024170 19791 tablet_service.cc:1514] Processing DeleteTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:03 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:48860
I20260925 10:16:24.024811 19866 tablet_replica.cc:330] stopping tablet replica
I20260925 10:16:24.024940 19866 raft_consensus.cc:2229] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:24.025025 19866 raft_consensus.cc:2258] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:24.025358 19866 ts_tablet_manager.cc:1905] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:24.027343 19866 ts_tablet_manager.cc:1918] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:16:24.027439 19866 log.cc:1196] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal/wals/d974ae86dc5447d9ab09fe08d72b0543
I20260925 10:16:24.027650 19866 ts_tablet_manager.cc:1939] T d974ae86dc5447d9ab09fe08d72b0543 P f8771dd2da4d46f0b0243e02340d7920: Deleting consensus metadata
I20260925 10:16:24.028254 19654 catalog_manager.cc:4841] TS f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193:34553): tablet d974ae86dc5447d9ab09fe08d72b0543 (table test-table [id=496acb589ff548389fa1146be976efe8]) successfully deleted
W20260925 10:16:24.092455 19861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:24.092830 19861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:24.092862 19861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:24.094168 19861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:24.094218 19861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:24.094252 19861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:16:24.095854 19861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:46645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=40927
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:24.096171 19861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:24.096418 19861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:24.101696 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:24.101678 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:24.101742 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:24.102020 19861 server_base.cc:1034] running on GCE node
I20260925 10:16:24.102242 19861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:24.102485 19861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:24.103734 19861 hybrid_clock.cc:648] HybridClock initialized: now 1790331384103717 us; error 24 us; skew 500 ppm
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331384, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:24.106559 19861 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:16:24.107035 19861 webserver.cc:460] Webserver started at http://127.3.152.194:40927/ using document root <none> and password file <none>
I20260925 10:16:24.107225 19861 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:24.107278 19861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:24.108701 19861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:24.109495 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.109683 19861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:24.109797 19861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "476e5a272ae34d7fa9881512075365d2"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "de5d02f5c1d6e42fe0c8e673c6ecfd3d"
server_key_iv: "94f46c838bb9496746a69b2f37c6e02a"
server_key_version: "encryptionkey@0"
I20260925 10:16:24.110196 19861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:24.140720 19861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:24.142917 19861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:24.143095 19861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:24.143347 19861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:24.143868 19888 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:16:24.145545 19861 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:16:24.145614 19861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:16:24.145680 19861 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:16:24.146379 19861 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:16:24.146440 19861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:16:24.146528 19888 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Bootstrap starting.
I20260925 10:16:24.153779 19861 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:46645
I20260925 10:16:24.153892 19995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:46645 every 8 connection(s)
I20260925 10:16:24.154234 19861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:16:24.156178 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 19861
I20260925 10:16:24.156482 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:38129
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=44403
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:16:24.156728 19888 log.cc:824] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:24.159652 19888 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:16:24.160089 19888 tablet_bootstrap.cc:492] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Bootstrap complete.
I20260925 10:16:24.161284 19888 ts_tablet_manager.cc:1397] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331384, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:24.162547 19888 raft_consensus.cc:348] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:24.162823 19888 raft_consensus.cc:729] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476e5a272ae34d7fa9881512075365d2, State: Initialized, Role: FOLLOWER
I20260925 10:16:24.162974 19888 consensus_queue.cc:260] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } } peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } }
I20260925 10:16:24.163312 19888 ts_tablet_manager.cc:1428] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:16:24.166134 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.155084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:55471 (local address 127.3.152.254:34129)
0925 10:16:24.155178 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.155181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.155360 (+ 179us) server_negotiation.cc:407] Connection header received
0925 10:16:24.156336 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.156337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.156353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.156390 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.157450 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.158261 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.159879 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.160180 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.160257 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.162728 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.162729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.162730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.162739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.164612 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.165009 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.165010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.165011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.165088 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.165423 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.165424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.165425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.165541 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.165596 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.165979 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.166068 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 10:16:24.166425 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.154955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:55471)
0925 10:16:24.155188 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.155246 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.156155 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.156606 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.156611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.157069 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:24.157315 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.157323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.158433 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.158433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.159719 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.159722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.160307 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.160309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.160465 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.161032 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:24.161043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:24.162619 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:24.164696 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.164702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.164703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.164923 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.165191 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.165192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.165192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.165349 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.165617 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:24.165624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:24.165811 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.165952 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.166154 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":122,"threads_started":1}
I20260925 10:16:24.167021 19996 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:24.167140 19996 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:24.167357 19996 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:24.168084 19669 ts_manager.cc:194] Registered new tserver with Master: 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
I20260925 10:16:24.168754 19669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:55471
I20260925 10:16:24.169777 19996 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.194@KRBTEST.COM
I20260925 10:16:24.176210 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.168542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.3.152.194:46645)
0925 10:16:24.168723 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.168727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.168737 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:16:24.168772 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.168774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.168790 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.168886 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.169167 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.169859 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.170685 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.171039 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.171119 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.173231 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.173239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.173239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.173245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.174926 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.175358 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.175361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.175361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.175414 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.175642 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.175642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.175642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.175801 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.175867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.176058 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.176139 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260925 10:16:24.176265 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.168430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.0.0.1:45362)
0925 10:16:24.168478 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.168488 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.168521 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.168890 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.168891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.169050 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:24.169089 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.169091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.169980 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.169981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.170573 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.170576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.171117 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.171118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.171184 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.171704 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:24.171709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:24.173131 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:24.175019 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.175021 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.175021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.175272 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.175493 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.175494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.175494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.175576 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.175870 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:24.175871 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:24.175926 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.176022 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.176150 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 10:16:24.176717 19930 tablet_service.cc:1514] Processing DeleteTablet for tablet d974ae86dc5447d9ab09fe08d72b0543 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:03 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:45362
I20260925 10:16:24.177383 20003 tablet_replica.cc:330] stopping tablet replica
I20260925 10:16:24.177505 20003 raft_consensus.cc:2229] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:24.177597 20003 raft_consensus.cc:2258] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:24.177917 20003 ts_tablet_manager.cc:1905] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:24.180120 20003 ts_tablet_manager.cc:1918] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:16:24.180207 20003 log.cc:1196] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/d974ae86dc5447d9ab09fe08d72b0543
I20260925 10:16:24.180449 20003 ts_tablet_manager.cc:1939] T d974ae86dc5447d9ab09fe08d72b0543 P 476e5a272ae34d7fa9881512075365d2: Deleting consensus metadata
I20260925 10:16:24.180985 19656 catalog_manager.cc:4841] TS 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): tablet d974ae86dc5447d9ab09fe08d72b0543 (table test-table [id=496acb589ff548389fa1146be976efe8]) successfully deleted
W20260925 10:16:24.249039 20000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:24.249259 20000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:24.249284 20000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:24.250388 20000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:24.250424 20000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:24.250450 20000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:16:24.252252 20000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:38129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=44403
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:24.252467 20000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:24.252676 20000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:24.257946 20008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:24.258049 20000 server_base.cc:1034] running on GCE node
W20260925 10:16:24.257942 20009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:24.257951 20011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:24.258419 20000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:24.258638 20000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:24.259828 20000 hybrid_clock.cc:648] HybridClock initialized: now 1790331384259803 us; error 29 us; skew 500 ppm
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331384, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:24.262441 20000 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:16:24.262854 20000 webserver.cc:460] Webserver started at http://127.3.152.195:44403/ using document root <none> and password file <none>
I20260925 10:16:24.263101 20000 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:24.263186 20000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:24.264679 20000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:16:24.265288 20018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.265455 20000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:16:24.265556 20000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "440606ab2d1ab44b9687beec7ad2f657"
server_key_iv: "d47d65f6be9a8ef0a1504bca2e2dcca0"
server_key_version: "encryptionkey@0"
I20260925 10:16:24.265951 20000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:24.273386 20000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:24.275359 20000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:24.275549 20000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:24.275800 20000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:24.276160 20000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:24.276214 20000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.276254 20000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:24.276279 20000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.282114 20000 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:38129
I20260925 10:16:24.282186 20131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:38129 every 8 connection(s)
I20260925 10:16:24.282554 20000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:16:24.286963 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20000
Sep 25 10:16:24 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331384, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:16:24.292474 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.283305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39371 (local address 127.3.152.254:34129)
0925 10:16:24.283395 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.283397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.283578 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:16:24.284108 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.284109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.284120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.284169 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.285079 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.285555 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.286514 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.286722 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.286776 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.289367 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.289368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.289369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.289375 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.290929 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.291455 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.291456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.291456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.291506 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.291789 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.291790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.291791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.291903 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.291994 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.292293 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.292373 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:16:24.292701 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.283257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:39371)
0925 10:16:24.283453 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.283489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.283955 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.284240 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.284243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.284811 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:24.285004 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.285008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.285633 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.285634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.286440 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.286442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.286809 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.286810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.286971 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.287624 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:24.287634 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:24.289277 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:24.291042 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.291062 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.291065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.291377 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.291584 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:24.291584 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:24.291585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:24.291716 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:24.292002 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:24.292009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:24.292152 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.292270 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.292453 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260925 10:16:24.293181 20132 heartbeater.cc:347] Connected to a master server at 127.3.152.254:34129
I20260925 10:16:24.293299 20132 heartbeater.cc:464] Registering TS with master...
I20260925 10:16:24.293515 20132 heartbeater.cc:511] Master 127.3.152.254:34129 requested a full tablet report, sending...
I20260925 10:16:24.293936 19669 ts_manager.cc:194] Registered new tserver with Master: e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129)
I20260925 10:16:24.294438 19669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:39371
I20260925 10:16:24.294771 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.287586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.3.152.254:34129)
0925 10:16:24.287706 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.287708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.287714 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:16:24.287730 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.287730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.287740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.287810 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.288564 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.289094 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.289956 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.290173 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.290250 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.291561 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.291563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.291563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.291570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.293521 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.293976 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.293977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.293977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.294045 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.294239 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.294239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.294240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.294358 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.294517 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.294559 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.294658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260925 10:16:24.295528 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:16:24.298101 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.295885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.3.152.254:34129)
0925 10:16:24.295921 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.295922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.296015 (+ 93us) server_negotiation.cc:407] Connection header received
0925 10:16:24.296091 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.296092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.296119 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.296178 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:16:24.296272 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.296776 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.297517 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.297725 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.297776 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.297901 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:16:24.297942 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 10:16:24.298059 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56280: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 10:16:24.298166 20136 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56280: token expired
W20260925 10:16:24.298354 19599 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.152.254:34129)
I20260925 10:16:24.303766 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.298625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.3.152.254:34129)
0925 10:16:24.298656 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.298658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.298665 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:16:24.298683 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.298684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.298692 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.298741 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:24.299155 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.299604 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.300316 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.300481 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.300560 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.301586 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:24.301587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:24.301587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:24.301593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:24.302962 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.303244 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.303245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.303245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.303292 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:24.303412 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:24.303413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:24.303413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:24.303495 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:24.303590 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.303648 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.303705 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 10:16:24.305034 19669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:16:24.306912 19669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:16:24.309012 19930 tablet_service.cc:1467] Processing CreateTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 (DEFAULT_TABLE table=test-table [id=7b7a6984474a416e861fbab0ed59fd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:24.309033 19791 tablet_service.cc:1467] Processing CreateTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 (DEFAULT_TABLE table=test-table [id=7b7a6984474a416e861fbab0ed59fd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:24.309213 19930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2bc29ec5c4bf59133af3894d326a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:24.309239 19791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2bc29ec5c4bf59133af3894d326a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:24.310791 19749 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap starting.
I20260925 10:16:24.310791 19888 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Bootstrap starting.
I20260925 10:16:24.311126 19888 tablet_bootstrap.cc:654] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:24.311143 19749 tablet_bootstrap.cc:654] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:24.312150 19888 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: No bootstrap required, opened a new log
I20260925 10:16:24.312202 19888 ts_tablet_manager.cc:1397] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:24.312212 19749 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: No bootstrap required, opened a new log
I20260925 10:16:24.312264 19749 ts_tablet_manager.cc:1397] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:24.312364 19888 raft_consensus.cc:348] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.312448 19888 raft_consensus.cc:374] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:24.312415 19749 raft_consensus.cc:348] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.312497 19888 raft_consensus.cc:729] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476e5a272ae34d7fa9881512075365d2, State: Initialized, Role: FOLLOWER
I20260925 10:16:24.312500 19749 raft_consensus.cc:374] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:24.312551 19749 raft_consensus.cc:729] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Initialized, Role: FOLLOWER
I20260925 10:16:24.312542 19888 consensus_queue.cc:260] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.312599 19749 consensus_queue.cc:260] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.312628 19888 ts_tablet_manager.cc:1428] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.312691 19749 ts_tablet_manager.cc:1428] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:24.314711 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.308773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38129 (local address 127.0.0.1:49804)
0925 10:16:24.308809 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.308820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.308860 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.311162 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.311164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.311165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:24.311217 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.311219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.312363 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.312364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.313485 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.313488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.314482 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.314483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.314540 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.314560 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.314612 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.314657 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 10:16:24.314749 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.310816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.3.152.195:38129)
0925 10:16:24.310999 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.311004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.311015 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:16:24.311059 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.311061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.311078 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.311160 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:24.311286 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.312258 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.313665 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.314417 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.314474 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.314517 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.314648 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.314679 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260925 10:16:24.315496 20066 tablet_service.cc:1467] Processing CreateTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 (DEFAULT_TABLE table=test-table [id=7b7a6984474a416e861fbab0ed59fd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:16:24.315829 20066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad2bc29ec5c4bf59133af3894d326a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:16:24.318982 20141 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Bootstrap starting.
I20260925 10:16:24.319591 20141 tablet_bootstrap.cc:654] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Neither blocks nor log segments found. Creating new log.
I20260925 10:16:24.320191 20141 log.cc:824] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:24.321015 20141 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: No bootstrap required, opened a new log
I20260925 10:16:24.321133 20141 ts_tablet_manager.cc:1397] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:16:24.322885 20141 raft_consensus.cc:348] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.323073 20141 raft_consensus.cc:374] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:16:24.323125 20141 raft_consensus.cc:729] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c0d330e9fa4a0bbbf34c4e2e2b5c36, State: Initialized, Role: FOLLOWER
I20260925 10:16:24.323268 20141 consensus_queue.cc:260] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.323611 20132 heartbeater.cc:503] Master 127.3.152.254:34129 was elected leader, sending a full tablet report...
I20260925 10:16:24.323619 20141 ts_tablet_manager.cc:1428] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:16:24.348186 19863 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:24.348346 19863 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.348650 19863 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
I20260925 10:16:24.352000 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.348914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:32879 (local address 127.3.152.194:46645)
0925 10:16:24.348957 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.348959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.348969 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:16:24.349051 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.349052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.349071 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.349133 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:24.349375 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.350031 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351123 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.351579 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351660 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.351756 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.351847 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.351892 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:16:24.352000 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.348844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.3.152.193:32879)
0925 10:16:24.348890 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.348902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.348957 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.349159 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.349160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.349162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:24.349264 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.349266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.350106 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.350107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.351022 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.351025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.351665 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.351747 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.351775 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.351826 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.351887 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 10:16:24.352020 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.348844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38129 (local address 127.3.152.193:42057)
0925 10:16:24.349251 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.349260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:24.349309 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:24.349473 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:24.349474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:24.349476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:16:24.349549 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.349550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.350222 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.350223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.351029 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:24.351032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.351727 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:24.351799 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.351803 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:16:24.351863 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:16:24.351900 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":188,"thread_start_us":122,"threads_started":1}
I20260925 10:16:24.352061 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.348907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:42057 (local address 127.3.152.195:38129)
0925 10:16:24.348957 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.348959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.349323 (+ 364us) server_negotiation.cc:407] Connection header received
0925 10:16:24.349414 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.349415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.349429 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.349468 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:16:24.349605 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.350140 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351161 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.351635 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.351714 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.351725 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.351927 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.351987 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:16:24.352342 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2bc29ec5c4bf59133af3894d326a0" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476e5a272ae34d7fa9881512075365d2" is_pre_election: true
I20260925 10:16:24.352440 20086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2bc29ec5c4bf59133af3894d326a0" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" is_pre_election: true
I20260925 10:16:24.352509 19950 raft_consensus.cc:2454] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 0.
I20260925 10:16:24.352610 20086 raft_consensus.cc:2454] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 0.
I20260925 10:16:24.352723 19746 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476e5a272ae34d7fa9881512075365d2, f8771dd2da4d46f0b0243e02340d7920; no voters:
I20260925 10:16:24.352862 19863 raft_consensus.cc:2781] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:16:24.352921 19863 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:16:24.352948 19863 raft_consensus.cc:3037] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:24.353610 19863 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.353778 19863 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 election: Requested vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
I20260925 10:16:24.353946 20086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2bc29ec5c4bf59133af3894d326a0" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
I20260925 10:16:24.354025 20086 raft_consensus.cc:3037] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:24.354154 19950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad2bc29ec5c4bf59133af3894d326a0" candidate_uuid: "f8771dd2da4d46f0b0243e02340d7920" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476e5a272ae34d7fa9881512075365d2"
I20260925 10:16:24.354250 19950 raft_consensus.cc:3037] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:16:24.355050 20086 raft_consensus.cc:2454] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 1.
I20260925 10:16:24.355191 19950 raft_consensus.cc:2454] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8771dd2da4d46f0b0243e02340d7920 in term 1.
I20260925 10:16:24.355259 19743 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2c0d330e9fa4a0bbbf34c4e2e2b5c36, f8771dd2da4d46f0b0243e02340d7920; no voters:
I20260925 10:16:24.355360 19863 raft_consensus.cc:2781] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:16:24.355504 19863 raft_consensus.cc:686] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 LEADER]: Becoming Leader. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Running, Role: LEADER
I20260925 10:16:24.355579 19863 consensus_queue.cc:237] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:24.356377 19669 catalog_manager.cc:5495] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "f8771dd2da4d46f0b0243e02340d7920" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } health_report { overall_health: UNKNOWN } } }
I20260925 10:16:24.372339 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.369773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.3.152.193:34553)
0925 10:16:24.369845 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:24.369847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:24.369857 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:16:24.369895 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:24.369896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:24.369913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:24.369975 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:16:24.370080 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.370663 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.371423 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:24.371622 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:24.371672 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:24.371910 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:16:24.372082 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:16:24.372170 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:24.372213 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 10:16:24.372248 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:16:24.373924 20086 raft_consensus.cc:1261] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Refusing update from remote peer f8771dd2da4d46f0b0243e02340d7920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:16:24.373924 19950 raft_consensus.cc:1261] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Refusing update from remote peer f8771dd2da4d46f0b0243e02340d7920: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:16:24.374325 19863 consensus_queue.cc:1035] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:16:24.374497 20145 consensus_queue.cc:1035] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:16:24.377121 20150 mvcc.cc:205] Tried to move back new op lower bound from 7333197350392999936 to 7333197350321020928. Current Snapshot: MvccSnapshot[applied={T|T < 7333197350392999936}]
I20260925 10:16:24.377355 20148 mvcc.cc:205] Tried to move back new op lower bound from 7333197350392999936 to 7333197350321020928. Current Snapshot: MvccSnapshot[applied={T|T < 7333197350392999936 or (T in {7333197350392999936})}]
I20260925 10:16:24.378616 19667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:16:24.378723 19667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:16:24.379457 19667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Sending DeleteTablet for 3 replicas of tablet dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:24.379743 19791 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:24 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:48860
I20260925 10:16:24.379776 19930 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:24 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:45362
I20260925 10:16:24.379840 19866 tablet_replica.cc:330] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: stopping tablet replica
I20260925 10:16:24.379810 20066 tablet_service.cc:1514] Processing DeleteTablet for tablet dad2bc29ec5c4bf59133af3894d326a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:16:24 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49804
I20260925 10:16:24.379879 20003 tablet_replica.cc:330] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: stopping tablet replica
I20260925 10:16:24.379892 19866 raft_consensus.cc:2229] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:16:24.379938 20003 raft_consensus.cc:2229] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:24.379998 19866 raft_consensus.cc:2258] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:24.380033 20003 raft_consensus.cc:2258] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:24.380090 20158 tablet_replica.cc:330] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: stopping tablet replica
I20260925 10:16:24.380200 20158 raft_consensus.cc:2229] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:16:24.380244 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19707
I20260925 10:16:24.380350 20158 raft_consensus.cc:2258] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:16:24.380364 19866 ts_tablet_manager.cc:1905] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:24.381228 20158 ts_tablet_manager.cc:1905] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:24.381887 20003 ts_tablet_manager.cc:1905] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:16:24.383055 20158 ts_tablet_manager.cc:1918] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:16:24.383145 20158 log.cc:1196] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:24.383445 20158 ts_tablet_manager.cc:1939] T dad2bc29ec5c4bf59133af3894d326a0 P e2c0d330e9fa4a0bbbf34c4e2e2b5c36: Deleting consensus metadata
I20260925 10:16:24.383598 20003 ts_tablet_manager.cc:1918] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:16:24.383661 20003 log.cc:1196] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:24.383872 20003 ts_tablet_manager.cc:1939] T dad2bc29ec5c4bf59133af3894d326a0 P 476e5a272ae34d7fa9881512075365d2: Deleting consensus metadata
I20260925 10:16:24.383934 19653 catalog_manager.cc:4841] TS e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): tablet dad2bc29ec5c4bf59133af3894d326a0 (table test-table [id=7b7a6984474a416e861fbab0ed59fd9a]) successfully deleted
I20260925 10:16:24.384205 19656 catalog_manager.cc:4841] TS 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): tablet dad2bc29ec5c4bf59133af3894d326a0 (table test-table [id=7b7a6984474a416e861fbab0ed59fd9a]) successfully deleted
W20260925 10:16:24.386365 19654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:34553 (error 108)
I20260925 10:16:24.386497 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19861
I20260925 10:16:24.391588 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20000
I20260925 10:16:24.391625 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:24.391439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34553 (local address 127.0.0.1:48872)
0925 10:16:24.391522 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:24.391582 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 10:16:24.391803 19654 catalog_manager.cc:4584] TS f8771dd2da4d46f0b0243e02340d7920 (127.3.152.193:34553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dad2bc29ec5c4bf59133af3894d326a0: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34553: connect: Connection refused (error 111)
I20260925 10:16:24.396246 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 19636
I20260925 10:16:44.402045 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--webserver_interface=127.3.152.254
--webserver_port=42379
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:34129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:44.485740 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:44.485992 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:44.486033 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:44.487154 20160 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 10:16:44.487186 20160 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 10:16:44.487198 20160 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 10:16:44.487210 20160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:16:44.487222 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:44.487236 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:44.487247 20160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:16:44.487259 20160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:16:44.488742 20160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:34129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:34129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:44.488960 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:44.489130 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:44.495925 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.495954 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.495929 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:44.496034 20160 server_base.cc:1034] running on GCE node
I20260925 10:16:44.496519 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:44.496832 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:44.498001 20160 hybrid_clock.cc:648] HybridClock initialized: now 1790331404497964 us; error 54 us; skew 500 ppm
Sep 25 10:16:44 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:44.504791 20160 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:16:44.505304 20160 webserver.cc:460] Webserver started at http://127.3.152.254:42379/ using document root <none> and password file <none>
I20260925 10:16:44.505535 20160 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:44.505610 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:44.507000 20160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:16:44.507759 20175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.507939 20160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:16:44.508077 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "d799418173904693954d9e1728608ae8"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "03b4b9f3a2bef5cfbd70e578e80ac16e"
server_key_iv: "50b58a9a0e0c95d1324cf6c54cbbcb9d"
server_key_version: "encryptionkey@0"
I20260925 10:16:44.508466 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:44.538678 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:44.541182 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:44.541355 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:44.545574 20160 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:34129
I20260925 10:16:44.545614 20227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:34129 every 8 connection(s)
I20260925 10:16:44.546025 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:16:44.547607 20228 sys_catalog.cc:263] Verifying existing consensus state
I20260925 10:16:44.548260 20228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap starting.
I20260925 10:16:44.548535 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20160
I20260925 10:16:44.548877 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:34553
--local_ip_for_outbound_sockets=127.3.152.193
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=42683
--webserver_interface=127.3.152.193
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:16:44.557811 20228 log.cc:824] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:44.560206 20228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:16:44.560554 20228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Bootstrap complete.
I20260925 10:16:44.562898 20228 raft_consensus.cc:348] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:44.563133 20228 raft_consensus.cc:729] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d799418173904693954d9e1728608ae8, State: Initialized, Role: FOLLOWER
I20260925 10:16:44.563323 20228 consensus_queue.cc:260] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:44.563385 20228 raft_consensus.cc:388] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:16:44.563441 20228 raft_consensus.cc:482] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:16:44.563489 20228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 10:16:44.564575 20228 raft_consensus.cc:504] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:44.564662 20228 leader_election.cc:302] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d799418173904693954d9e1728608ae8; no voters:
I20260925 10:16:44.564879 20228 leader_election.cc:288] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 10:16:44.564921 20232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 10:16:44.565035 20232 raft_consensus.cc:686] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [term 3 LEADER]: Becoming Leader. State: Replica: d799418173904693954d9e1728608ae8, State: Running, Role: LEADER
I20260925 10:16:44.565124 20232 consensus_queue.cc:237] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } }
I20260925 10:16:44.565167 20228 sys_catalog.cc:564] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:16:44.565395 20232 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d799418173904693954d9e1728608ae8. Latest consensus state: current_term: 3 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:44.565470 20232 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:16:44.565402 20233 sys_catalog.cc:455] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d799418173904693954d9e1728608ae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d799418173904693954d9e1728608ae8" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 34129 } } }
I20260925 10:16:44.565531 20233 sys_catalog.cc:458] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8 [sys.catalog]: This master's current role is: LEADER
I20260925 10:16:44.565797 20236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:16:44.566644 20236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:16:44.566764 20236 catalog_manager.cc:1259] Loaded cluster ID: c9272cffeb8f4cdeb59c3e641fa8f58a
I20260925 10:16:44.566802 20236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:16:44.568008 20236 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:16:44.568193 20236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Loaded TSK: 0
I20260925 10:16:44.568248 20236 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d799418173904693954d9e1728608ae8: Loaded TSK: 1
F20260925 10:16:44.568490 20236 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8dcd5119ad google::LogMessage::Fail() at ??:0
@ 0x7f8dcd51488a google::LogMessage::SendToLog() at ??:0
@ 0x7f8dcd5114de google::LogMessage::Flush() at ??:0
@ 0x7f8dcd512a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8dcfc7e104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8dcfc7bd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8dd546b1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8dd546b48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8dd546b594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8dd5431c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8dd543bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8dcf3df0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8dcf3d4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8dd02386db start_thread at ??:0
@ 0x7f8dcc00f61f clone at ??:0
W20260925 10:16:44.653179 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:44.653386 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:44.653415 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:44.654567 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:44.654613 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:44.654644 20230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:16:44.656176 20230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:34553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=42683
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:44.656427 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:44.656667 20230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:44.661904 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.661922 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.661993 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:44.661981 20230 server_base.cc:1034] running on GCE node
I20260925 10:16:44.662320 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:44.662567 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:44.663728 20230 hybrid_clock.cc:648] HybridClock initialized: now 1790331404663716 us; error 30 us; skew 500 ppm
Sep 25 10:16:44 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:44.666447 20230 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:16:44.666952 20230 webserver.cc:460] Webserver started at http://127.3.152.193:42683/ using document root <none> and password file <none>
I20260925 10:16:44.667203 20230 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:44.667271 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:44.668691 20230 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:16:44.669449 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.669632 20230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:44.669725 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "f8771dd2da4d46f0b0243e02340d7920"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "084385f93205ebc12e4dfe9538541260"
server_key_iv: "46595d0a7254bf78d83d0dcec63c6c4c"
server_key_version: "encryptionkey@0"
I20260925 10:16:44.670143 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:44.713230 20230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:44.715857 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:44.716053 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:44.716361 20230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:44.716863 20286 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 10:16:44.718408 20230 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 10:16:44.718472 20230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 10:16:44.718508 20230 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 10:16:44.719400 20230 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 10:16:44.719446 20230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 10:16:44.719540 20286 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap starting.
I20260925 10:16:44.721396 20286 log.cc:824] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Log is configured to *not* fsync() on all Append() calls
I20260925 10:16:44.724681 20286 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 10:16:44.725076 20286 tablet_bootstrap.cc:492] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Bootstrap complete.
I20260925 10:16:44.725258 20286 ts_tablet_manager.cc:1397] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 10:16:44.726610 20230 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34553
I20260925 10:16:44.726914 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34553 every 8 connection(s)
I20260925 10:16:44.726996 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:16:44.727559 20286 raft_consensus.cc:348] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:44.727908 20286 raft_consensus.cc:729] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8771dd2da4d46f0b0243e02340d7920, State: Initialized, Role: FOLLOWER
I20260925 10:16:44.728075 20286 consensus_queue.cc:260] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:44.728379 20286 ts_tablet_manager.cc:1428] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:16:44.735284 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20230
I20260925 10:16:44.735638 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:46645
--local_ip_for_outbound_sockets=127.3.152.194
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=40927
--webserver_interface=127.3.152.194
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:44.775859 20398 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:16:44.728227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:36939)
0925 10:16:44.728468 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.728521 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:44.728658 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:44.775652 (+ 46994us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":70,"threads_started":1}
W20260925 10:16:44.776163 20395 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:34129 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:34129: Client connection negotiation failed: client connection to 127.3.152.254:34129: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 10:16:44.776611 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.776433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:42275)
0925 10:16:44.776489 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.776587 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 10:16:44.776911 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.776848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:34749)
0925 10:16:44.776875 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.776896 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 10:16:44.777014 20395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 10:16:44.821504 20399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:44.821816 20399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:44.821859 20399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:44.823030 20399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:44.823096 20399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:44.823139 20399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:16:44.824688 20399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:46645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=40927
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:44.824944 20399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:44.825174 20399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:44.830261 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.830287 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:44.830544 20399 server_base.cc:1034] running on GCE node
W20260925 10:16:44.830271 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:44.830803 20399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:44.831038 20399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:44.832175 20399 hybrid_clock.cc:648] HybridClock initialized: now 1790331404832162 us; error 29 us; skew 500 ppm
Sep 25 10:16:44 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:44.834656 20399 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:16:44.835088 20399 webserver.cc:460] Webserver started at http://127.3.152.194:40927/ using document root <none> and password file <none>
I20260925 10:16:44.835323 20399 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:44.835391 20399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:44.836769 20399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:44.837364 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.837531 20399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:16:44.837615 20399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "476e5a272ae34d7fa9881512075365d2"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "de5d02f5c1d6e42fe0c8e673c6ecfd3d"
server_key_iv: "94f46c838bb9496746a69b2f37c6e02a"
server_key_version: "encryptionkey@0"
I20260925 10:16:44.837965 20399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:44.851681 20399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:44.854039 20399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:44.854244 20399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:44.854486 20399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:44.854795 20399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:44.854848 20399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.854887 20399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:44.854918 20399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.860991 20399 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:46645
I20260925 10:16:44.861086 20527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:46645 every 8 connection(s)
I20260925 10:16:44.861397 20399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:16:44.862433 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.862037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:58797)
0925 10:16:44.862179 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.862293 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260925 10:16:44.862790 20528 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:34129 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:34129: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
I20260925 10:16:44.863067 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.862974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:44631)
0925 10:16:44.863023 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.863051 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:16:44.863343 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.863256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:48413)
0925 10:16:44.863297 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.863328 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 10:16:44.863444 20528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:16:44.871235 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20399
I20260925 10:16:44.871515 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:38129
--local_ip_for_outbound_sockets=127.3.152.195
--tserver_master_addrs=127.3.152.254:34129
--webserver_port=44403
--webserver_interface=127.3.152.195
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:16:44.952939 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:16:44.953157 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:16:44.953183 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:16:44.954312 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:16:44.954350 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:16:44.954397 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:16:44.955895 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:38129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=44403
--tserver_master_addrs=127.3.152.254:34129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:16:44.956184 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:16:44.956396 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:16:44.961356 20539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.961378 20537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:16:44.961365 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:16:44.961692 20531 server_base.cc:1034] running on GCE node
I20260925 10:16:44.961913 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:16:44.962177 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:16:44.963349 20531 hybrid_clock.cc:648] HybridClock initialized: now 1790331404963341 us; error 36 us; skew 500 ppm
Sep 25 10:16:44 dist-test-slave-dzwq krb5kdc[19089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:16:44.965952 20531 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:16:44.966470 20531 webserver.cc:460] Webserver started at http://127.3.152.195:44403/ using document root <none> and password file <none>
I20260925 10:16:44.966697 20531 fs_manager.cc:359] Metadata directory not provided
I20260925 10:16:44.966760 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:16:44.968271 20531 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:44.968911 20546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.969100 20531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:16:44.969194 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36"
format_stamp: "Formatted at 2026-09-25 10:16:03 on dist-test-slave-dzwq"
server_key: "440606ab2d1ab44b9687beec7ad2f657"
server_key_iv: "d47d65f6be9a8ef0a1504bca2e2dcca0"
server_key_version: "encryptionkey@0"
I20260925 10:16:44.969530 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:16:44.979657 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:16:44.982035 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:16:44.982162 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:16:44.982343 20531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:16:44.982671 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:16:44.982712 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.982734 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:16:44.982746 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:16:44.988621 20531 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:38129
I20260925 10:16:44.988695 20659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:38129 every 8 connection(s)
I20260925 10:16:44.989094 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:16:44.990172 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.989785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:56903)
0925 10:16:44.989946 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.990022 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20260925 10:16:44.990607 20660 heartbeater.cc:644] Failed to heartbeat to 127.3.152.254:34129 (0 consecutive failures): Network error: Failed to ping master at 127.3.152.254:34129: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
I20260925 10:16:44.990895 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.990809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:49065)
0925 10:16:44.990852 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.990878 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 10:16:44.991164 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:44.991093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:60421)
0925 10:16:44.991130 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:44.991151 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 10:16:44.991274 20660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 10:16:44.996768 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20531
I20260925 10:16:45.777633 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:45.777386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:55297)
0925 10:16:45.777516 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:45.777561 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260925 10:16:45.864114 20665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:45.863863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:38267)
0925 10:16:45.864024 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:45.864064 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260925 10:16:45.991940 20666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:45.991637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:48205)
0925 10:16:45.991848 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:45.991891 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":56,"threads_started":1}
I20260925 10:16:46.256275 20667 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:46.256440 20667 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:46.256816 20667 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
Sep 25 10:16:46 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.195@KRBTEST.COM
Sep 25 10:16:46 dist-test-slave-dzwq krb5kdc[19089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331404, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.194@KRBTEST.COM
I20260925 10:16:46.267783 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.257174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:37479 (local address 127.3.152.195:38129)
0925 10:16:46.257351 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:46.257365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:46.257401 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:16:46.258009 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:46.258012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:46.258077 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:46.258238 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:46.259129 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.260071 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261041 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.261456 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:46.264018 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:46.264022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:46.264023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:46.264034 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:46.266071 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266500 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:46.266503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:46.266503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:46.266595 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266918 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:46.266919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:46.266919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:46.267163 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:46.267246 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:46.267535 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:16:46.267668 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":51,"threads_started":1}
I20260925 10:16:46.267791 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.257233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:40871 (local address 127.3.152.194:46645)
0925 10:16:46.257365 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 10:16:46.257371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:16:46.257401 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:16:46.258009 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:16:46.258012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:16:46.258090 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:16:46.258238 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:16:46.259251 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.260259 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261067 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.261454 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261571 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:46.264411 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:16:46.264414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:16:46.264414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:16:46.264424 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:16:46.266239 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266584 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:46.266587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:46.266587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:46.266680 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266918 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:16:46.266919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:16:46.266920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:16:46.267162 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:16:46.267246 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:16:46.267527 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 10:16:46.267659 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":49,"threads_started":1}
I20260925 10:16:46.267849 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.257039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38129 (local address 127.3.152.193:37479)
0925 10:16:46.257105 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:46.257121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:46.257203 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:46.258304 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:46.258307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:46.258741 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:46.259002 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:46.259009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.260185 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.260186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:46.260953 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:46.260955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.261545 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:46.261680 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:46.262304 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:46.262308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:46.263874 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:46.266171 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:46.266180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:46.266389 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:46.266731 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:46.266733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:46.266855 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:46.267237 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:46.267244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:46.267369 (+ 125us) client_negotiation.cc:769] Sending connection context
0925 10:16:46.267469 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:16:46.267662 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":4}
I20260925 10:16:46.267869 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.257039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:46645 (local address 127.3.152.193:40871)
0925 10:16:46.257169 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:46.257181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:16:46.257250 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:16:46.258368 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:16:46.258370 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:16:46.258742 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:16:46.259139 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:46.259141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.260351 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.260352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:46.260958 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:16:46.260960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:16:46.261545 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:16:46.261546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:16:46.261680 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:16:46.262247 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:16:46.262256 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:16:46.264300 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:16:46.266313 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:46.266314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:46.266483 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:46.266752 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:16:46.266753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:16:46.266754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:16:46.266855 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:16:46.267240 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:16:46.267246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:16:46.267369 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 10:16:46.267470 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 10:16:46.267738 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":146,"thread_start_us":52,"threads_started":1}
W20260925 10:16:46.268684 20280 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
W20260925 10:16:46.269279 20283 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:46.269330 20283 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8771dd2da4d46f0b0243e02340d7920; no voters: 476e5a272ae34d7fa9881512075365d2, e2c0d330e9fa4a0bbbf34c4e2e2b5c36
I20260925 10:16:46.269469 20667 raft_consensus.cc:2726] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:16:46.778400 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.778207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:33993)
0925 10:16:46.778315 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:46.778352 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260925 10:16:46.864789 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.864557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:33353)
0925 10:16:46.864701 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:46.864748 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260925 10:16:46.992734 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:46.992463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:37211)
0925 10:16:46.992634 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:46.992689 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260925 10:16:47.779111 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:47.778867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:38819)
0925 10:16:47.779008 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:47.779043 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260925 10:16:47.865507 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:47.865268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:43989)
0925 10:16:47.865408 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:47.865454 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260925 10:16:47.993518 20676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:47.993277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:53575)
0925 10:16:47.993429 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:47.993472 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260925 10:16:48.344373 20677 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:48.344477 20677 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:48.344686 20677 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
W20260925 10:16:48.345127 20280 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
W20260925 10:16:48.345245 20283 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:48.345283 20283 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8771dd2da4d46f0b0243e02340d7920; no voters: 476e5a272ae34d7fa9881512075365d2, e2c0d330e9fa4a0bbbf34c4e2e2b5c36
I20260925 10:16:48.345362 20677 raft_consensus.cc:2726] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:16:48.779901 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:48.779663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:46429)
0925 10:16:48.779803 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:48.779838 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260925 10:16:48.866214 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:48.865979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:50001)
0925 10:16:48.866119 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:48.866163 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260925 10:16:48.994287 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:48.994068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:51287)
0925 10:16:48.994200 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:48.994242 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260925 10:16:49.780766 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:49.780506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:37803)
0925 10:16:49.780648 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:49.780686 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 10:16:49.866904 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:49.866708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:40519)
0925 10:16:49.866827 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:49.866858 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 10:16:49.995052 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:49.994824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:41411)
0925 10:16:49.994967 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:49.995007 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260925 10:16:50.781564 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:50.781351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:47165)
0925 10:16:50.781484 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:50.781519 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260925 10:16:50.867789 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:50.867562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:37235)
0925 10:16:50.867699 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:50.867742 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260925 10:16:50.995874 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:50.995637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:58177)
0925 10:16:50.995794 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:50.995827 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260925 10:16:51.268642 20687 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:51.268733 20687 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:51.268896 20687 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
W20260925 10:16:51.269289 20283 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
W20260925 10:16:51.269356 20280 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:51.269433 20280 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8771dd2da4d46f0b0243e02340d7920; no voters: 476e5a272ae34d7fa9881512075365d2, e2c0d330e9fa4a0bbbf34c4e2e2b5c36
I20260925 10:16:51.269524 20687 raft_consensus.cc:2726] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:16:51.782264 20688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:51.782054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:32923)
0925 10:16:51.782180 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:51.782217 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260925 10:16:51.868481 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:51.868268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:32967)
0925 10:16:51.868397 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:51.868437 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260925 10:16:51.996546 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:51.996324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:44969)
0925 10:16:51.996461 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:51.996500 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260925 10:16:52.783115 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:52.782897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:34317)
0925 10:16:52.783031 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:52.783067 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260925 10:16:52.869141 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:52.868912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:34005)
0925 10:16:52.869032 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:52.869089 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260925 10:16:52.997327 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:52.997072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:53777)
0925 10:16:52.997225 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:52.997277 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260925 10:16:53.783846 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:53.783589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:48977)
0925 10:16:53.783757 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:53.783798 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260925 10:16:53.869915 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:53.869687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:44319)
0925 10:16:53.869827 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:53.869867 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260925 10:16:53.998135 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:53.997893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:46391)
0925 10:16:53.998041 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:53.998085 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260925 10:16:54.784682 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:54.784399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:48105)
0925 10:16:54.784567 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:54.784612 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260925 10:16:54.870699 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:54.870433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:56579)
0925 10:16:54.870602 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:54.870649 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260925 10:16:54.998926 20699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:54.998686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:45165)
0925 10:16:54.998838 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:54.998880 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260925 10:16:55.785349 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:55.785118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:39295)
0925 10:16:55.785263 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:55.785299 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260925 10:16:55.871371 20701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:55.871119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:60781)
0925 10:16:55.871279 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:55.871321 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
I20260925 10:16:55.999799 20702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:55.999473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:60077)
0925 10:16:55.999704 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:55.999750 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":130,"threads_started":1}
I20260925 10:16:56.060389 20703 raft_consensus.cc:482] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:16:56.060525 20703 raft_consensus.cc:504] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c0d330e9fa4a0bbbf34c4e2e2b5c36" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38129 } } peers { permanent_uuid: "f8771dd2da4d46f0b0243e02340d7920" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34553 } } peers { permanent_uuid: "476e5a272ae34d7fa9881512075365d2" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 46645 } }
I20260925 10:16:56.060745 20703 leader_election.cc:288] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129), 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645)
W20260925 10:16:56.061132 20280 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2c0d330e9fa4a0bbbf34c4e2e2b5c36 (127.3.152.195:38129): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
W20260925 10:16:56.061208 20283 leader_election.cc:341] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 476e5a272ae34d7fa9881512075365d2 (127.3.152.194:46645): Not found: Tablet not found: dad2bc29ec5c4bf59133af3894d326a0
I20260925 10:16:56.061269 20283 leader_election.cc:302] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8771dd2da4d46f0b0243e02340d7920; no voters: 476e5a272ae34d7fa9881512075365d2, e2c0d330e9fa4a0bbbf34c4e2e2b5c36
I20260925 10:16:56.061344 20703 raft_consensus.cc:2726] T dad2bc29ec5c4bf59133af3894d326a0 P f8771dd2da4d46f0b0243e02340d7920 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 10:16:56.786091 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:56.785870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:57977)
0925 10:16:56.786001 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:56.786044 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 10:16:56.872314 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:56.871949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:56427)
0925 10:16:56.872164 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:56.872253 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":140,"threads_started":1}
I20260925 10:16:57.000535 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:57.000286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:33045)
0925 10:16:57.000438 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:57.000478 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260925 10:16:57.786870 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:57.786615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:36139)
0925 10:16:57.786782 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:57.786822 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260925 10:16:57.873044 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:57.872831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:59813)
0925 10:16:57.872953 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:57.872995 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260925 10:16:58.001479 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:58.001192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:55439)
0925 10:16:58.001380 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:58.001425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260925 10:16:58.787640 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:58.787409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:39073)
0925 10:16:58.787548 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:58.787590 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260925 10:16:58.873728 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:58.873511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:45163)
0925 10:16:58.873625 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:58.873679 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260925 10:16:59.002208 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:59.001994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:42515)
0925 10:16:59.002122 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:59.002155 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260925 10:16:59.788571 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:59.788346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.193:54035)
0925 10:16:59.788471 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:59.788520 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 10:16:59.874552 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:16:59.874308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.194:55341)
0925 10:16:59.874442 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 10:16:59.874482 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260925 10:17:00.003345 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:00.003067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:34129 (local address 127.3.152.195:50909)
0925 10:17:00.003250 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:00.003296 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.254:34129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260925 10:17:00.004710 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20230
I20260925 10:17:00.009717 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20399
I20260925 10:17:00.014147 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20531
2026-09-25T10:17:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58268 ms)
[----------] 30 tests from SecurityITest (261002 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:00 dist-test-slave-dzwq krb5kdc[20718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:00 dist-test-slave-dzwq krb5kdc[20718](info): set up 2 sockets
Sep 25 10:17:00 dist-test-slave-dzwq krb5kdc[20718](info): commencing operation
krb5kdc: starting...
W20260925 10:17:01.060405 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:01.188063 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41511
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:01.272943 20731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:01.273195 20731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:01.273226 20731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:01.274498 20731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:01.274546 20731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:01.274564 20731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:01.274581 20731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:01.274596 20731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:01.276343 20731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:01.276580 20731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:01.276777 20731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:01.282166 20736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.282433 20731 server_base.cc:1034] running on GCE node
W20260925 10:17:01.282173 20737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:01.282173 20739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.282790 20731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:01.283051 20731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:01.284243 20731 hybrid_clock.cc:648] HybridClock initialized: now 1790331421284233 us; error 32 us; skew 500 ppm
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:01.286813 20731 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:01.287297 20731 webserver.cc:460] Webserver started at http://127.3.152.254:41439/ using document root <none> and password file <none>
I20260925 10:17:01.287526 20731 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:01.287595 20731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:01.287748 20731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:01.288906 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "26848974e8b041729410a8174f6212be"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "8154824ebe95d13c49b1751d1f9d43ba"
server_key_iv: "298b9bb7d22652d8823d9674e943d9bf"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.289289 20731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "26848974e8b041729410a8174f6212be"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "8154824ebe95d13c49b1751d1f9d43ba"
server_key_iv: "298b9bb7d22652d8823d9674e943d9bf"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.290616 20731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:17:01.291355 20746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.291523 20731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:17:01.291627 20731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "26848974e8b041729410a8174f6212be"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "8154824ebe95d13c49b1751d1f9d43ba"
server_key_iv: "298b9bb7d22652d8823d9674e943d9bf"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.291700 20731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:01.312595 20731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:01.315043 20731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:01.315235 20731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:01.319650 20731 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41511
I20260925 10:17:01.319700 20798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41511 every 8 connection(s)
I20260925 10:17:01.320119 20731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:01.321151 20799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:01.323342 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20731
I20260925 10:17:01.323485 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:01.323592 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: Bootstrap starting.
I20260925 10:17:01.323717 3683 external_mini_cluster.cc:1428] Setting key ab7ea86494bffb16639b5f3735b76990
I20260925 10:17:01.324378 20799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:01.324718 20799 log.cc:824] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:01.325426 20799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: No bootstrap required, opened a new log
I20260925 10:17:01.327315 20799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } }
I20260925 10:17:01.327476 20799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:01.327536 20799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26848974e8b041729410a8174f6212be, State: Initialized, Role: FOLLOWER
I20260925 10:17:01.327669 20799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } }
I20260925 10:17:01.327754 20799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:01.327797 20799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:01.327847 20799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:01.328564 20799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } }
I20260925 10:17:01.328679 20799 leader_election.cc:302] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26848974e8b041729410a8174f6212be; no voters:
I20260925 10:17:01.328874 20799 leader_election.cc:288] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:01.328940 20804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:01.329104 20804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [term 1 LEADER]: Becoming Leader. State: Replica: 26848974e8b041729410a8174f6212be, State: Running, Role: LEADER
I20260925 10:17:01.329154 20799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:01.329298 20804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } }
I20260925 10:17:01.329715 20804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26848974e8b041729410a8174f6212be. Latest consensus state: current_term: 1 leader_uuid: "26848974e8b041729410a8174f6212be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } } }
I20260925 10:17:01.329828 20806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26848974e8b041729410a8174f6212be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26848974e8b041729410a8174f6212be" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41511 } } }
I20260925 10:17:01.329874 20804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:01.329931 20806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:01.330205 20812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:01.330775 20812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:01.332453 20812 catalog_manager.cc:1347] Generated new cluster ID: 4f75aaaa43974754829e884ca40eabf4
I20260925 10:17:01.332506 20812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:01.336037 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.324343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.3.152.254:41511)
0925 10:17:01.324531 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.324540 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.324576 (+ 36us) server_negotiation.cc:407] Connection header received
0925 10:17:01.325171 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.325181 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.325249 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.325398 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:01.326063 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.326962 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.327884 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.328558 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.328683 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.331739 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:01.331743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:01.331743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:01.331755 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:01.333851 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.334288 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.334292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.334292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.334416 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.334650 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.334650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.334651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.334962 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:01.335122 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.335641 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.335785 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260925 10:17:01.340572 20812 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:01.341285 20812 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:01.347606 20812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: Generated new TSK 0
I20260925 10:17:01.347805 20812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:17:01.379042 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:01.462827 20825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:01.463047 20825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:01.463073 20825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:01.464334 20825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:01.464381 20825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:01.464409 20825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:01.465818 20825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:01.466012 20825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:01.466189 20825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:01.471196 20833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:01.471196 20830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.471442 20825 server_base.cc:1034] running on GCE node
W20260925 10:17:01.471256 20831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.471735 20825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:01.472074 20825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:01.473269 20825 hybrid_clock.cc:648] HybridClock initialized: now 1790331421473253 us; error 47 us; skew 500 ppm
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:01.475895 20825 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:17:01.476389 20825 webserver.cc:460] Webserver started at http://127.3.152.193:35537/ using document root <none> and password file <none>
I20260925 10:17:01.476639 20825 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:01.476712 20825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:01.476857 20825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:01.477980 20825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "fe29f82528dd495a841b1e5fa417d897"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "ab5e05031ef3bfbeef1c06903ad792a0"
server_key_iv: "92af1ea4ad9da1cdb47ba765d2236eb1"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.478379 20825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "fe29f82528dd495a841b1e5fa417d897"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "ab5e05031ef3bfbeef1c06903ad792a0"
server_key_iv: "92af1ea4ad9da1cdb47ba765d2236eb1"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.479717 20825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:01.480521 20840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.480700 20825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.480814 20825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "fe29f82528dd495a841b1e5fa417d897"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "ab5e05031ef3bfbeef1c06903ad792a0"
server_key_iv: "92af1ea4ad9da1cdb47ba765d2236eb1"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.480895 20825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:01.508287 20825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:01.510694 20825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:01.510878 20825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:01.511149 20825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:01.511475 20825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:01.511528 20825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.511569 20825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:01.511624 20825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.517692 20825 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:44955
I20260925 10:17:01.517735 20953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:44955 every 8 connection(s)
I20260925 10:17:01.518069 20825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:17:01.524446 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20825
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:01.524578 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:01.524816 3683 external_mini_cluster.cc:1428] Setting key 81742f2934d99594c5362cba10fdb88a
I20260925 10:17:01.528144 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.518816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:59641 (local address 127.3.152.254:41511)
0925 10:17:01.518903 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.518906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.519086 (+ 180us) server_negotiation.cc:407] Connection header received
0925 10:17:01.519572 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.519619 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.519630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.519686 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:01.520761 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.521321 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.522327 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.522557 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.522627 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.525127 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:01.525129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:01.525129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:01.525139 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:01.526674 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.527119 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.527120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.527121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.527183 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.527444 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.527445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.527445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.527566 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:01.527645 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.527934 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.528051 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 10:17:01.528348 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.518766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41511 (local address 127.3.152.193:59641)
0925 10:17:01.518958 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.518997 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.519483 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.519780 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.519784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.520414 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:01.520620 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.520626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.521406 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.521406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.522231 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.522233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.522646 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.522647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.522762 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.523312 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:01.523320 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:01.525031 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:01.526781 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.526790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.526793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.527038 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.527255 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.527256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.527256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.527375 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.527638 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:01.527647 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:01.527794 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.527902 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.528097 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260925 10:17:01.528865 20954 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41511
I20260925 10:17:01.528986 20954 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:01.529223 20954 heartbeater.cc:511] Master 127.3.152.254:41511 requested a full tablet report, sending...
I20260925 10:17:01.529899 20763 ts_manager.cc:194] Registered new tserver with Master: fe29f82528dd495a841b1e5fa417d897 (127.3.152.193:44955)
I20260925 10:17:01.530951 20763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:59641
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:17:01.551455 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:01.634477 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:01.634693 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:01.634720 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:01.635952 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:01.636018 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:01.636044 20959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:01.637566 20959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:01.637830 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:01.638080 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:01.643195 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:01.643195 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.643523 20959 server_base.cc:1034] running on GCE node
W20260925 10:17:01.643258 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.643836 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:01.644120 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:01.645287 20959 hybrid_clock.cc:648] HybridClock initialized: now 1790331421645275 us; error 30 us; skew 500 ppm
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:01.647780 20959 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:17:01.648260 20959 webserver.cc:460] Webserver started at http://127.3.152.194:40913/ using document root <none> and password file <none>
I20260925 10:17:01.648456 20959 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:01.648496 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:01.648586 20959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:01.649574 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "0e65f6de7f394880aef1d39da74ef759"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "12f91d737a9c25e211f3c776be0684a0"
server_key_iv: "f75dfd867667ea8e563b6b3345fb2422"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.649878 20959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "0e65f6de7f394880aef1d39da74ef759"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "12f91d737a9c25e211f3c776be0684a0"
server_key_iv: "f75dfd867667ea8e563b6b3345fb2422"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.651176 20959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:01.651888 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.652113 20959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:01.652200 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "0e65f6de7f394880aef1d39da74ef759"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "12f91d737a9c25e211f3c776be0684a0"
server_key_iv: "f75dfd867667ea8e563b6b3345fb2422"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.652254 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:01.666890 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:01.668972 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:01.669090 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:01.669276 20959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:01.669626 20959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:01.669660 20959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.669683 20959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:01.669731 20959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.675405 20959 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:41237
I20260925 10:17:01.675484 21087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:41237 every 8 connection(s)
I20260925 10:17:01.675817 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:01.677675 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 20959
I20260925 10:17:01.677794 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:01.677981 3683 external_mini_cluster.cc:1428] Setting key 38d3375950b60fc83bd9ed5c942cae8a
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:01.686712 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.676655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:38763 (local address 127.3.152.254:41511)
0925 10:17:01.676748 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.676750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.676931 (+ 181us) server_negotiation.cc:407] Connection header received
0925 10:17:01.677497 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.677499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.677512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.677559 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:01.678844 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.679351 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.680461 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.680673 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.680771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.683174 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:01.683176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:01.683177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:01.683186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:01.685150 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.685641 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.685643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.685644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.685708 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.685971 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.685972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.685973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.686135 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:01.686218 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.686484 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.686573 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 10:17:01.686830 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.676630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41511 (local address 127.3.152.194:38763)
0925 10:17:01.676804 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.676848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.677379 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.677628 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.677632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.678471 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:01.678763 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.678771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.679449 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.679450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.680368 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.680370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.680784 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.680785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.680920 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.681437 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:01.681447 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:01.683080 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:01.685257 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.685264 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.685266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.685555 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.685791 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.685792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.685792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.685899 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.686212 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:01.686218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:01.686340 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.686430 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.686619 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:17:01.687350 21088 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41511
I20260925 10:17:01.687462 21088 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:01.687659 21088 heartbeater.cc:511] Master 127.3.152.254:41511 requested a full tablet report, sending...
I20260925 10:17:01.688108 20763 ts_manager.cc:194] Registered new tserver with Master: 0e65f6de7f394880aef1d39da74ef759 (127.3.152.194:41237)
I20260925 10:17:01.688553 20763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:38763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:17:01.707618 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:41817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:01.793501 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:01.793772 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:01.793809 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:01.794981 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:01.795060 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:01.795104 21093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:01.796520 21093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:01.796721 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:01.796892 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:01.802026 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:01.802042 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:01.802069 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:01.802347 21093 server_base.cc:1034] running on GCE node
I20260925 10:17:01.802554 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:01.802794 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:01.803930 21093 hybrid_clock.cc:648] HybridClock initialized: now 1790331421803909 us; error 42 us; skew 500 ppm
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:01.806519 21093 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:17:01.806993 21093 webserver.cc:460] Webserver started at http://127.3.152.195:36509/ using document root <none> and password file <none>
I20260925 10:17:01.807224 21093 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:01.807271 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:01.807418 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:01.808586 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "0b6435f421d045b58cd75c7b4807c3a6"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "f8313acf8a8af6e401d3b5e648e6dc38"
server_key_iv: "4b2f4e64e06d667316730148b2d4672d"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.808934 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "0b6435f421d045b58cd75c7b4807c3a6"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "f8313acf8a8af6e401d3b5e648e6dc38"
server_key_iv: "4b2f4e64e06d667316730148b2d4672d"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.810293 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:01.811142 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.811327 21093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:01.811429 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "0b6435f421d045b58cd75c7b4807c3a6"
format_stamp: "Formatted at 2026-09-25 10:17:01 on dist-test-slave-dzwq"
server_key: "f8313acf8a8af6e401d3b5e648e6dc38"
server_key_iv: "4b2f4e64e06d667316730148b2d4672d"
server_key_version: "encryptionkey@0"
I20260925 10:17:01.811512 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:01.821076 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:01.823148 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:01.823343 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:01.823598 21093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:01.823912 21093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:01.823966 21093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.824054 21093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:01.824079 21093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:01.830267 21093 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:34293
I20260925 10:17:01.830314 21221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:34293 every 8 connection(s)
I20260925 10:17:01.830641 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:17:01.832753 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21093
I20260925 10:17:01.832890 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:01.833088 3683 external_mini_cluster.cc:1428] Setting key d21b10e5a0a0dcce2bf99fcc62ccf612
Sep 25 10:17:01 dist-test-slave-dzwq krb5kdc[20718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:01.840248 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.831362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:59445 (local address 127.3.152.254:41511)
0925 10:17:01.831407 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.831409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.831669 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:17:01.832291 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.832292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.832305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.832357 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:01.833649 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.834161 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.835189 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.835403 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.835489 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.837669 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:01.837670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:01.837670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:01.837677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:01.838878 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.839237 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.839238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.839238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.839287 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.839526 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.839527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.839527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.839661 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:01.839741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.840046 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.840130 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:01.840426 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.831392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:41511 (local address 127.3.152.195:59445)
0925 10:17:01.831543 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.831581 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.832182 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.832417 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.832421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.833274 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:01.833546 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.833560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.834258 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.834259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.835074 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.835076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.835501 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.835501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.835634 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.836229 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:01.836237 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:01.837578 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:01.838950 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.838957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.838959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.839172 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.839344 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:01.839345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:01.839345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:01.839441 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:01.839743 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:01.839751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:01.839881 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.840014 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.840180 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260925 10:17:01.840915 21222 heartbeater.cc:347] Connected to a master server at 127.3.152.254:41511
I20260925 10:17:01.841017 21222 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:01.841168 21222 heartbeater.cc:511] Master 127.3.152.254:41511 requested a full tablet report, sending...
I20260925 10:17:01.841542 20763 ts_manager.cc:194] Registered new tserver with Master: 0b6435f421d045b58cd75c7b4807c3a6 (127.3.152.195:34293)
I20260925 10:17:01.841917 20763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:59445
I20260925 10:17:01.844825 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:01.851892 20802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.846161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.3.152.254:41511)
0925 10:17:01.846218 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.846220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.846322 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:17:01.846362 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.846363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.846372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.846420 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:01.846922 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.847391 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.848373 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.848570 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.848636 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.849803 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:01.849804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:01.849804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:01.849809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:01.850964 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.851267 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.851268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.851268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.851316 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:01.851444 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:01.851445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:01.851445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:01.851575 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:01.851664 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.851741 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.851798 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:17:01.853653 20763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:01.855319 20763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:01.862212 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.858093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.3.152.194:41237)
0925 10:17:01.858266 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.858270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.858362 (+ 92us) server_negotiation.cc:407] Connection header received
0925 10:17:01.858404 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.858406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.858422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.858591 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:01.858915 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.860031 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861230 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.861736 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861835 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.861896 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.862057 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.862103 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260925 10:17:01.862228 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.857836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:41237 (local address 127.0.0.1:60378)
0925 10:17:01.858285 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.858294 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.858343 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.858757 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.858757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.858760 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:01.858802 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.858803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.860158 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.860159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.861140 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.861142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.861841 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.861926 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.861943 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.862037 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.862096 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":80,"threads_started":1}
I20260925 10:17:01.862241 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.857820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:34293 (local address 127.0.0.1:57564)
0925 10:17:01.858165 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.858175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.858218 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.858590 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.858593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.858596 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:01.858672 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.858675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.859597 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.859598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.860883 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.860886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.861931 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.862010 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.862015 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.862071 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.862116 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":97,"threads_started":1}
I20260925 10:17:01.862426 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.858062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.3.152.195:34293)
0925 10:17:01.858350 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.858353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.858362 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:01.858404 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.858406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.858422 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.858516 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:01.858756 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.859494 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.860994 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.861755 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861834 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.861893 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.862284 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.862332 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260925 10:17:01.862744 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.857816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44955 (local address 127.0.0.1:45794)
0925 10:17:01.857991 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.858032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.858102 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.858677 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.858678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.858680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:01.858758 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.858761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.860160 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.860161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.861364 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.861368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.862500 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.862501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.862567 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.862572 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.862632 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.862681 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260925 10:17:01.863032 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.857918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.3.152.193:44955)
0925 10:17:01.858103 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.858109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.858120 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:01.858249 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.858252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.858271 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.858355 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:01.858840 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.860047 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.861569 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.862321 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.862393 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.862444 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.862885 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.862940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260925 10:17:01.863354 21156 tablet_service.cc:1467] Processing CreateTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 (DEFAULT_TABLE table=test-table [id=b05b1abc9bce40079e69f2fc0a84686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:01.863719 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35d8e9a564241a2a5993e878f8a5a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:01.863692 20888 tablet_service.cc:1467] Processing CreateTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 (DEFAULT_TABLE table=test-table [id=b05b1abc9bce40079e69f2fc0a84686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:01.864079 20888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35d8e9a564241a2a5993e878f8a5a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:01.866638 21239 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Bootstrap starting.
I20260925 10:17:01.867035 21240 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: Bootstrap starting.
I20260925 10:17:01.867246 21022 tablet_service.cc:1467] Processing CreateTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 (DEFAULT_TABLE table=test-table [id=b05b1abc9bce40079e69f2fc0a84686e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:01.867398 21239 tablet_bootstrap.cc:654] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:01.867602 21022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c35d8e9a564241a2a5993e878f8a5a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:01.867707 21240 tablet_bootstrap.cc:654] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:01.868029 21239 log.cc:824] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:01.868448 21240 log.cc:824] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:01.868922 21239 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: No bootstrap required, opened a new log
I20260925 10:17:01.869010 21239 ts_tablet_manager.cc:1397] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:17:01.869236 21240 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: No bootstrap required, opened a new log
I20260925 10:17:01.869323 21240 ts_tablet_manager.cc:1397] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:01.870994 21243 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Bootstrap starting.
I20260925 10:17:01.870824 21240 raft_consensus.cc:348] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.871065 21240 raft_consensus.cc:374] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:01.871132 21240 raft_consensus.cc:729] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe29f82528dd495a841b1e5fa417d897, State: Initialized, Role: FOLLOWER
I20260925 10:17:01.871174 21239 raft_consensus.cc:348] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.871336 21239 raft_consensus.cc:374] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:01.871328 21240 consensus_queue.cc:260] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.871398 21239 raft_consensus.cc:729] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6435f421d045b58cd75c7b4807c3a6, State: Initialized, Role: FOLLOWER
I20260925 10:17:01.871527 21239 consensus_queue.cc:260] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.871714 21240 ts_tablet_manager.cc:1428] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:17:01.871762 20954 heartbeater.cc:503] Master 127.3.152.254:41511 was elected leader, sending a full tablet report...
I20260925 10:17:01.871820 21243 tablet_bootstrap.cc:654] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:01.871883 21239 ts_tablet_manager.cc:1428] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:17:01.871927 21222 heartbeater.cc:503] Master 127.3.152.254:41511 was elected leader, sending a full tablet report...
I20260925 10:17:01.872452 21243 log.cc:824] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:01.873350 21243 tablet_bootstrap.cc:492] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: No bootstrap required, opened a new log
I20260925 10:17:01.873410 21243 ts_tablet_manager.cc:1397] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:01.875033 21243 raft_consensus.cc:348] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.875174 21243 raft_consensus.cc:374] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:01.875200 21243 raft_consensus.cc:729] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e65f6de7f394880aef1d39da74ef759, State: Initialized, Role: FOLLOWER
I20260925 10:17:01.875340 21243 consensus_queue.cc:260] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.875664 21243 ts_tablet_manager.cc:1428] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:01.875664 21088 heartbeater.cc:503] Master 127.3.152.254:41511 was elected leader, sending a full tablet report...
I20260925 10:17:01.889858 21245 raft_consensus.cc:482] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:01.890019 21245 raft_consensus.cc:504] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.890435 21245 leader_election.cc:288] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe29f82528dd495a841b1e5fa417d897 (127.3.152.193:44955), 0e65f6de7f394880aef1d39da74ef759 (127.3.152.194:41237)
I20260925 10:17:01.894393 21224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.890644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:41237 (local address 127.3.152.195:37001)
0925 10:17:01.890712 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.890724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.890771 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.890951 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.890952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.890954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:01.891058 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.891059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.892030 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.892031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.893295 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.893298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.894081 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.894082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.894192 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.894196 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.894250 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.894302 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":33}
I20260925 10:17:01.894577 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.890697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:37001 (local address 127.3.152.194:41237)
0925 10:17:01.890745 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.890748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.890759 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:01.890843 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.890848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.890858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.890931 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:01.891134 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.891921 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.893388 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.893991 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.894062 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.894071 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.894444 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.894497 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:17:01.894542 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.890727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:47851 (local address 127.3.152.193:44955)
0925 10:17:01.890769 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.890771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.890905 (+ 134us) server_negotiation.cc:407] Connection header received
0925 10:17:01.890967 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.890968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.890977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.891028 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:01.891192 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.891929 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.893388 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.893996 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.894063 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.894071 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.894341 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.894424 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:17:01.894403 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.890657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44955 (local address 127.3.152.195:47851)
0925 10:17:01.890826 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.890841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:01.890894 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:01.891050 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:01.891050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:01.891051 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:01.891123 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.891124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.892030 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.892031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.893295 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:01.893298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.894126 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.894127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:01.894192 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.894197 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:01.894289 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:17:01.894319 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":30,"thread_start_us":123,"threads_started":1}
I20260925 10:17:01.895186 20908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35d8e9a564241a2a5993e878f8a5a85" candidate_uuid: "0b6435f421d045b58cd75c7b4807c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe29f82528dd495a841b1e5fa417d897" is_pre_election: true
I20260925 10:17:01.895386 20908 raft_consensus.cc:2454] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6435f421d045b58cd75c7b4807c3a6 in term 0.
I20260925 10:17:01.895345 21042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35d8e9a564241a2a5993e878f8a5a85" candidate_uuid: "0b6435f421d045b58cd75c7b4807c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e65f6de7f394880aef1d39da74ef759" is_pre_election: true
I20260925 10:17:01.895519 21042 raft_consensus.cc:2454] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6435f421d045b58cd75c7b4807c3a6 in term 0.
I20260925 10:17:01.895664 21109 leader_election.cc:302] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6435f421d045b58cd75c7b4807c3a6, fe29f82528dd495a841b1e5fa417d897; no voters:
I20260925 10:17:01.895820 21245 raft_consensus.cc:2781] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:01.895879 21245 raft_consensus.cc:482] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:01.895905 21245 raft_consensus.cc:3037] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:01.896677 21245 raft_consensus.cc:504] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.896832 21245 leader_election.cc:288] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [CANDIDATE]: Term 1 election: Requested vote from peers fe29f82528dd495a841b1e5fa417d897 (127.3.152.193:44955), 0e65f6de7f394880aef1d39da74ef759 (127.3.152.194:41237)
I20260925 10:17:01.896991 20908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35d8e9a564241a2a5993e878f8a5a85" candidate_uuid: "0b6435f421d045b58cd75c7b4807c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe29f82528dd495a841b1e5fa417d897"
I20260925 10:17:01.897023 21042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c35d8e9a564241a2a5993e878f8a5a85" candidate_uuid: "0b6435f421d045b58cd75c7b4807c3a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e65f6de7f394880aef1d39da74ef759"
I20260925 10:17:01.897089 20908 raft_consensus.cc:3037] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:01.897094 21042 raft_consensus.cc:3037] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:01.897821 21042 raft_consensus.cc:2454] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6435f421d045b58cd75c7b4807c3a6 in term 1.
I20260925 10:17:01.897967 20908 raft_consensus.cc:2454] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6435f421d045b58cd75c7b4807c3a6 in term 1.
I20260925 10:17:01.897998 21111 leader_election.cc:302] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6435f421d045b58cd75c7b4807c3a6, 0e65f6de7f394880aef1d39da74ef759; no voters:
I20260925 10:17:01.898074 21245 raft_consensus.cc:2781] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:01.898257 21245 raft_consensus.cc:686] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6435f421d045b58cd75c7b4807c3a6, State: Running, Role: LEADER
I20260925 10:17:01.898363 21245 consensus_queue.cc:237] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } }
I20260925 10:17:01.899209 20763 catalog_manager.cc:5495] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6435f421d045b58cd75c7b4807c3a6 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "0b6435f421d045b58cd75c7b4807c3a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6435f421d045b58cd75c7b4807c3a6" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 34293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 } health_report { overall_health: UNKNOWN } } }
I20260925 10:17:01.908152 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.905690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.3.152.195:34293)
0925 10:17:01.905758 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:01.905760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:01.905769 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:01.905811 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:01.905812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:01.905828 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:01.905900 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:01.906056 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.906530 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.907320 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:01.907516 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:01.907592 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:01.907780 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:01.907881 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:01.907969 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:01.908020 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 10:17:01.908040 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:17:01.910010 21042 raft_consensus.cc:1261] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 1 FOLLOWER]: Refusing update from remote peer 0b6435f421d045b58cd75c7b4807c3a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:01.910010 20908 raft_consensus.cc:1261] T c35d8e9a564241a2a5993e878f8a5a85 P fe29f82528dd495a841b1e5fa417d897 [term 1 FOLLOWER]: Refusing update from remote peer 0b6435f421d045b58cd75c7b4807c3a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:01.910476 21249 consensus_queue.cc:1035] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe29f82528dd495a841b1e5fa417d897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:01.910668 21245 consensus_queue.cc:1035] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e65f6de7f394880aef1d39da74ef759" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:01.913743 21254 mvcc.cc:205] Tried to move back new op lower bound from 7333197504140537856 to 7333197504096395264. Current Snapshot: MvccSnapshot[applied={T|T < 7333197504140537856}]
I20260925 10:17:01.915491 20762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:01.915575 20762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:01.916328 20762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26848974e8b041729410a8174f6212be: Sending DeleteTablet for 3 replicas of tablet c35d8e9a564241a2a5993e878f8a5a85
I20260925 10:17:01.916661 20888 tablet_service.cc:1514] Processing DeleteTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:01 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:45794
I20260925 10:17:01.916673 21022 tablet_service.cc:1514] Processing DeleteTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:01 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:60378
I20260925 10:17:01.916874 21156 tablet_service.cc:1514] Processing DeleteTablet for tablet c35d8e9a564241a2a5993e878f8a5a85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:01 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:57564
I20260925 10:17:01.916949 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20825
I20260925 10:17:01.917755 21263 tablet_replica.cc:330] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: stopping tablet replica
I20260925 10:17:01.917878 21263 raft_consensus.cc:2229] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:01.917994 21263 raft_consensus.cc:2258] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:01.918135 21264 tablet_replica.cc:330] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: stopping tablet replica
I20260925 10:17:01.918228 21264 raft_consensus.cc:2229] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:01.918347 21264 raft_consensus.cc:2258] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:01.918591 21263 ts_tablet_manager.cc:1905] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:01.918792 21264 ts_tablet_manager.cc:1905] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:01.920861 21264 ts_tablet_manager.cc:1918] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:01.920861 21263 ts_tablet_manager.cc:1918] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:01.920941 21263 log.cc:1196] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/c35d8e9a564241a2a5993e878f8a5a85
I20260925 10:17:01.920941 21264 log.cc:1196] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/c35d8e9a564241a2a5993e878f8a5a85
I20260925 10:17:01.921236 21263 ts_tablet_manager.cc:1939] T c35d8e9a564241a2a5993e878f8a5a85 P 0e65f6de7f394880aef1d39da74ef759: Deleting consensus metadata
I20260925 10:17:01.921236 21264 ts_tablet_manager.cc:1939] T c35d8e9a564241a2a5993e878f8a5a85 P 0b6435f421d045b58cd75c7b4807c3a6: Deleting consensus metadata
I20260925 10:17:01.921667 20750 catalog_manager.cc:4841] TS 0b6435f421d045b58cd75c7b4807c3a6 (127.3.152.195:34293): tablet c35d8e9a564241a2a5993e878f8a5a85 (table test-table [id=b05b1abc9bce40079e69f2fc0a84686e]) successfully deleted
I20260925 10:17:01.921746 20749 catalog_manager.cc:4841] TS 0e65f6de7f394880aef1d39da74ef759 (127.3.152.194:41237): tablet c35d8e9a564241a2a5993e878f8a5a85 (table test-table [id=b05b1abc9bce40079e69f2fc0a84686e]) successfully deleted
W20260925 10:17:01.927157 20747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:44955 (error 108)
I20260925 10:17:01.927387 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20959
I20260925 10:17:01.929560 21233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:01.929373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44955 (local address 127.0.0.1:45798)
0925 10:17:01.929437 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:01.929508 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:17:01.929731 20747 catalog_manager.cc:4584] TS fe29f82528dd495a841b1e5fa417d897 (127.3.152.193:44955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c35d8e9a564241a2a5993e878f8a5a85: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44955: connect: Connection refused (error 111)
I20260925 10:17:01.933002 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21093
I20260925 10:17:01.937769 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 20731
2026-09-25T10:17:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1923 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T10:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:01Z Disabled control of system clock
I20260925 10:17:01.966219 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35787
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:35787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:02.047355 21269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:02.047581 21269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:02.047601 21269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:02.048823 21269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:02.048869 21269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:02.048882 21269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:02.048895 21269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:02.048909 21269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:02.050367 21269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:35787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:35787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:02.050633 21269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:02.050951 21269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:02.056119 21275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.056072 21274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.056072 21277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:02.056145 21269 server_base.cc:1034] running on GCE node
I20260925 10:17:02.056671 21269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:02.056981 21269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:02.058151 21269 hybrid_clock.cc:648] HybridClock initialized: now 1790331422058121 us; error 48 us; skew 500 ppm
I20260925 10:17:02.059435 21269 webserver.cc:460] Webserver started at http://127.3.152.254:36355/ using document root <none> and password file <none>
I20260925 10:17:02.059767 21269 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:02.059844 21269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:02.060052 21269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:02.061199 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feffa18be98a0c56fe4e4dc2c162fc0e"
server_key_iv: "b9e762d604cca867ed4c29be298c9743"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.061579 21269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feffa18be98a0c56fe4e4dc2c162fc0e"
server_key_iv: "b9e762d604cca867ed4c29be298c9743"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.062927 21269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:02.063772 21283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.064047 21269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:02.064167 21269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feffa18be98a0c56fe4e4dc2c162fc0e"
server_key_iv: "b9e762d604cca867ed4c29be298c9743"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.064262 21269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:02.083663 21269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:02.084183 21269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:02.084364 21269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:02.088920 21269 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:35787
I20260925 10:17:02.088953 21335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:35787 every 8 connection(s)
I20260925 10:17:02.089366 21269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:02.090427 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:02.090909 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21269
I20260925 10:17:02.091121 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:02.091369 3683 external_mini_cluster.cc:1428] Setting key d4d58ba1c3a0267cd46467e8eb48d624
I20260925 10:17:02.093837 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: Bootstrap starting.
I20260925 10:17:02.094558 21336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:02.094866 21336 log.cc:824] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:02.095593 21336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: No bootstrap required, opened a new log
I20260925 10:17:02.098052 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.092123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.3.152.254:35787)
0925 10:17:02.092357 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.092367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.092402 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:17:02.092982 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.092991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.093078 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.093245 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:02.093438 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.094797 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.095658 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.096332 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.096462 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.097259 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:02.097264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:02.097265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:02.097475 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:02.097520 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:02.097537 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:02.097614 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.097704 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.097798 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260925 10:17:02.098357 21336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } }
I20260925 10:17:02.098544 21336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:02.098579 21336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea1115e71c146ceb1ba9cba2ae9ab6f, State: Initialized, Role: FOLLOWER
I20260925 10:17:02.098694 21336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } }
I20260925 10:17:02.098798 21336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:02.098838 21336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:02.098884 21336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:02.099443 21336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } }
I20260925 10:17:02.099566 21336 leader_election.cc:302] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ea1115e71c146ceb1ba9cba2ae9ab6f; no voters:
I20260925 10:17:02.099771 21336 leader_election.cc:288] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:02.099876 21341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:02.100155 21336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:02.100164 21341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [term 1 LEADER]: Becoming Leader. State: Replica: 0ea1115e71c146ceb1ba9cba2ae9ab6f, State: Running, Role: LEADER
I20260925 10:17:02.100373 21341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } }
I20260925 10:17:02.100752 21342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } } }
I20260925 10:17:02.100772 21343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea1115e71c146ceb1ba9cba2ae9ab6f. Latest consensus state: current_term: 1 leader_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ea1115e71c146ceb1ba9cba2ae9ab6f" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 35787 } } }
I20260925 10:17:02.100834 21342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:02.100905 21343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:02.101110 21350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:02.101677 21350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:02.103204 21350 catalog_manager.cc:1347] Generated new cluster ID: 92f0c660a09a4e528ffa71f07cda0be8
I20260925 10:17:02.103272 21350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:02.120692 21350 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:02.121285 21350 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:02.132258 21350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: Generated new TSK 0
I20260925 10:17:02.132602 21350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:02.134100 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--builtin_ntp_servers=127.3.152.212:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:02.237279 21360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:02.237542 21360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:02.237586 21360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:02.238765 21360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:02.238835 21360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:02.238880 21360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:02.240516 21360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:02.240790 21360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:02.241122 21360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:02.247071 21365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.247095 21366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.247129 21368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:02.247545 21360 server_base.cc:1034] running on GCE node
I20260925 10:17:02.247766 21360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:02.248090 21360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:02.249253 21360 hybrid_clock.cc:648] HybridClock initialized: now 1790331422249236 us; error 33 us; skew 500 ppm
I20260925 10:17:02.250825 21360 webserver.cc:460] Webserver started at http://127.3.152.193:38971/ using document root <none> and password file <none>
I20260925 10:17:02.251107 21360 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:02.251188 21360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:02.251356 21360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:02.252578 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "498b426105854f329f125665bfa82d97"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "b91e477731b15e6bdfb91ad2680fd486"
server_key_iv: "af31922f8ee7e9aed7f7a08d059bbf6f"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.253021 21360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "498b426105854f329f125665bfa82d97"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "b91e477731b15e6bdfb91ad2680fd486"
server_key_iv: "af31922f8ee7e9aed7f7a08d059bbf6f"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.254590 21360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:02.255461 21374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.255725 21360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:02.255849 21360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "498b426105854f329f125665bfa82d97"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "b91e477731b15e6bdfb91ad2680fd486"
server_key_iv: "af31922f8ee7e9aed7f7a08d059bbf6f"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.255934 21360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:02.268288 21360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:02.268628 21360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:02.268790 21360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:02.269061 21360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:02.269419 21360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:02.269475 21360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.269544 21360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:02.269577 21360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.275393 21360 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:44261
I20260925 10:17:02.275445 21487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:44261 every 8 connection(s)
I20260925 10:17:02.275827 21360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:17:02.280225 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21360
I20260925 10:17:02.280355 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:02.280571 3683 external_mini_cluster.cc:1428] Setting key 93346d5d1b9b7441f59330f84225feac
I20260925 10:17:02.281471 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.276639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:33521 (local address 127.3.152.254:35787)
0925 10:17:02.276688 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.276689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.276949 (+ 260us) server_negotiation.cc:407] Connection header received
0925 10:17:02.277506 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.277508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.277526 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.277591 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:02.278273 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.278799 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.279794 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.280059 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.280133 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.281152 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:02.281153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:02.281153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:02.281158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:02.281175 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:02.281179 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:02.281241 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.281359 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.281392 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:17:02.281615 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--builtin_ntp_servers=127.3.152.212:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:17:02.281728 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.276622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35787 (local address 127.3.152.193:33521)
0925 10:17:02.276824 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.276859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.277396 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.277670 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.277674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.277682 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:02.278168 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.278172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.278902 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.278904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.279700 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.279703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.280154 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.280155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.280321 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.280989 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:02.280989 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:02.281019 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:02.281043 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:02.281252 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:02.281259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:02.281262 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.281346 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.281451 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260925 10:17:02.282189 21488 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35787
I20260925 10:17:02.282305 21488 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:02.282501 21488 heartbeater.cc:511] Master 127.3.152.254:35787 requested a full tablet report, sending...
I20260925 10:17:02.283062 21300 ts_manager.cc:194] Registered new tserver with Master: 498b426105854f329f125665bfa82d97 (127.3.152.193:44261)
I20260925 10:17:02.283741 21300 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.193:33521
W20260925 10:17:02.363745 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:02.364020 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:02.364068 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:02.365275 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:02.365317 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:02.365343 21491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:02.366813 21491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:02.367072 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:02.367404 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:02.372517 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.372577 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:02.372777 21491 server_base.cc:1034] running on GCE node
W20260925 10:17:02.372531 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:02.373072 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:02.373329 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:02.374519 21491 hybrid_clock.cc:648] HybridClock initialized: now 1790331422374508 us; error 40 us; skew 500 ppm
I20260925 10:17:02.375902 21491 webserver.cc:460] Webserver started at http://127.3.152.194:46119/ using document root <none> and password file <none>
I20260925 10:17:02.376248 21491 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:02.376329 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:02.376492 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:02.377663 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "555a3b0678334410811710ca09f40648"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feb8a8f009316611195d9ffc10fd2d21"
server_key_iv: "75c2b1bd7e7bb4707f22cfaf484ec6db"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.378059 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "555a3b0678334410811710ca09f40648"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feb8a8f009316611195d9ffc10fd2d21"
server_key_iv: "75c2b1bd7e7bb4707f22cfaf484ec6db"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.379410 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:02.380168 21505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.380359 21491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:02.380477 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "555a3b0678334410811710ca09f40648"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "feb8a8f009316611195d9ffc10fd2d21"
server_key_iv: "75c2b1bd7e7bb4707f22cfaf484ec6db"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.380558 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:02.393618 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:02.393966 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:02.394131 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:02.394411 21491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:02.394776 21491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:02.394832 21491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.394876 21491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:02.394901 21491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.401111 21491 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:38719
I20260925 10:17:02.401165 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:38719 every 8 connection(s)
I20260925 10:17:02.401542 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:02.407373 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21491
I20260925 10:17:02.407483 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:02.407740 3683 external_mini_cluster.cc:1428] Setting key d49282da231b4c3b3377b5d63ad7070b
I20260925 10:17:02.407856 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.402363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:51445 (local address 127.3.152.254:35787)
0925 10:17:02.402438 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.402440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.402645 (+ 205us) server_negotiation.cc:407] Connection header received
0925 10:17:02.403136 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.403137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.403147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.403201 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:02.403891 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.404486 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.405614 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.406052 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.406160 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.407448 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:02.407450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:02.407451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:02.407458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:02.407479 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:02.407484 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:02.407570 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.407731 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.407770 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 10:17:02.408306 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.402358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35787 (local address 127.3.152.194:51445)
0925 10:17:02.402536 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.402571 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.403046 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.403266 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.403269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.403277 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:02.403785 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.403789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.404643 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.404644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.405462 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.405464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.406270 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.406271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.406460 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.407213 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:02.407214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:02.407261 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:02.407295 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:02.407619 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:02.407628 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:02.407632 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.408017 (+ 385us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.408131 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260925 10:17:02.408782 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--builtin_ntp_servers=127.3.152.212:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:17:02.408856 21619 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35787
I20260925 10:17:02.408948 21619 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:02.409143 21619 heartbeater.cc:511] Master 127.3.152.254:35787 requested a full tablet report, sending...
I20260925 10:17:02.409531 21300 ts_manager.cc:194] Registered new tserver with Master: 555a3b0678334410811710ca09f40648 (127.3.152.194:38719)
I20260925 10:17:02.409920 21300 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.194:51445
W20260925 10:17:02.491616 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:02.491894 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:02.491932 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:02.493099 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:02.493168 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:02.493212 21622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:02.494597 21622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:35787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:02.494887 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:02.495203 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 10:17:02.500349 21622 server_base.cc:1034] running on GCE node
W20260925 10:17:02.500322 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.500330 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:02.500322 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:02.500871 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:02.501125 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:02.502288 21622 hybrid_clock.cc:648] HybridClock initialized: now 1790331422502284 us; error 43 us; skew 500 ppm
I20260925 10:17:02.503713 21622 webserver.cc:460] Webserver started at http://127.3.152.195:45511/ using document root <none> and password file <none>
I20260925 10:17:02.504024 21622 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:02.504108 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:02.504261 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:02.505419 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "6d9f68ca622a422c8e258bfd1a733680"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "2c974dc51309cea06a747d361e7bf951"
server_key_iv: "4643a3242992be611f1960f7a184458b"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.505792 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "6d9f68ca622a422c8e258bfd1a733680"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "2c974dc51309cea06a747d361e7bf951"
server_key_iv: "4643a3242992be611f1960f7a184458b"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.507112 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:02.507942 21636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.508221 21622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:02.508322 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "6d9f68ca622a422c8e258bfd1a733680"
format_stamp: "Formatted at 2026-09-25 10:17:02 on dist-test-slave-dzwq"
server_key: "2c974dc51309cea06a747d361e7bf951"
server_key_iv: "4643a3242992be611f1960f7a184458b"
server_key_version: "encryptionkey@0"
I20260925 10:17:02.508409 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:02.544625 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:02.544916 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:02.545030 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:02.545208 21622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:02.545589 21622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:02.545655 21622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.545691 21622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:02.545746 21622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:02.551738 21622 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:46219
I20260925 10:17:02.551792 21749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:46219 every 8 connection(s)
I20260925 10:17:02.552191 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:17:02.554191 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21622
I20260925 10:17:02.554316 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:02.554538 3683 external_mini_cluster.cc:1428] Setting key 06bd67ef3923e48a405e571c3451d37b
I20260925 10:17:02.558511 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.553043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39321 (local address 127.3.152.254:35787)
0925 10:17:02.553135 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.553138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.553348 (+ 210us) server_negotiation.cc:407] Connection header received
0925 10:17:02.553814 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.553815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.553825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.553896 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:02.554883 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.555672 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.556747 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.556984 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.557054 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.558183 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:02.558185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:02.558185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:02.558190 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:02.558207 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:02.558212 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:02.558270 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.558397 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.558427 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 10:17:02.558852 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.553017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:35787 (local address 127.3.152.195:39321)
0925 10:17:02.553196 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.553254 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.553728 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.553952 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.553955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.553963 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:02.554771 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.554780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.555756 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.555757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.556648 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.556650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.557103 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.557104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.557324 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.558004 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:02.558004 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:02.558053 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:02.558086 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:02.558281 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:02.558286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:02.558292 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.558382 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.558582 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 10:17:02.559332 21750 heartbeater.cc:347] Connected to a master server at 127.3.152.254:35787
I20260925 10:17:02.559432 21750 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:02.559615 21750 heartbeater.cc:511] Master 127.3.152.254:35787 requested a full tablet report, sending...
I20260925 10:17:02.560052 21300 ts_manager.cc:194] Registered new tserver with Master: 6d9f68ca622a422c8e258bfd1a733680 (127.3.152.195:46219)
I20260925 10:17:02.560425 21300 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.195:39321
I20260925 10:17:02.566458 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:02.570912 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.568015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.3.152.254:35787)
0925 10:17:02.568079 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.568082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.568220 (+ 138us) server_negotiation.cc:407] Connection header received
0925 10:17:02.568264 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.568264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.568275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.568330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:02.568510 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.569004 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.569758 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.569940 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.569999 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.570629 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:02.570630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:02.570630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:02.570634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:02.570648 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:02.570651 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:02.570703 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.570762 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.570782 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:17:02.572484 21300 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:02.573964 21300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:02.580462 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46219 (local address 127.0.0.1:53680)
0925 10:17:02.576378 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.576422 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.576492 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.576960 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.576963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.576965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:02.577029 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.577031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.578029 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.578030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.579233 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.579235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.580164 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.580165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.580245 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.580260 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.580325 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.580380 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260925 10:17:02.580711 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.3.152.195:46219)
0925 10:17:02.576789 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.576794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.576804 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:02.576840 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.576842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.576856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.576957 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:02.577114 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.577939 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.579321 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.580076 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.580164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.580262 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.580577 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.580622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":73,"threads_started":1}
I20260925 10:17:02.580739 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.3.152.193:44261)
0925 10:17:02.576779 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.576783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.576793 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:02.576836 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.576838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.576856 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.576943 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:02.577170 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.578325 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.579515 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.580252 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.580330 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.580414 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.580500 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.580653 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260925 10:17:02.580999 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44261 (local address 127.0.0.1:44694)
0925 10:17:02.576593 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.576603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.576656 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.577012 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.577013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.577015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:02.577077 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.577078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.578423 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.578424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.579430 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.579431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.580335 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.580336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.580404 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.580408 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.580475 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.580522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":68,"threads_started":1}
I20260925 10:17:02.581476 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38719 (local address 127.0.0.1:53114)
0925 10:17:02.576696 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.576705 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.576758 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.577067 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.577067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.577069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:02.577100 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.577101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.579497 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.579498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.580256 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.580258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.581255 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.581256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.581316 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.581320 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.581371 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.581413 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":96,"threads_started":1}
I20260925 10:17:02.581380 21684 tablet_service.cc:1467] Processing CreateTablet for tablet 5af03fc398c246d899844da46f981c25 (DEFAULT_TABLE table=test-table [id=706e7c809df443349707ce60c94bfaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:02.581750 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.576529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.3.152.194:38719)
0925 10:17:02.576706 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.576710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.576753 (+ 43us) server_negotiation.cc:407] Connection header received
0925 10:17:02.576836 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.576838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.576856 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.576943 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:02.577194 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.578330 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.580358 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.581065 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.581130 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.581219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.581620 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.581662 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260925 10:17:02.581771 21684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af03fc398c246d899844da46f981c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:02.581736 21422 tablet_service.cc:1467] Processing CreateTablet for tablet 5af03fc398c246d899844da46f981c25 (DEFAULT_TABLE table=test-table [id=706e7c809df443349707ce60c94bfaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:02.582093 21422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af03fc398c246d899844da46f981c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:02.582423 21553 tablet_service.cc:1467] Processing CreateTablet for tablet 5af03fc398c246d899844da46f981c25 (DEFAULT_TABLE table=test-table [id=706e7c809df443349707ce60c94bfaf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:02.582734 21553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af03fc398c246d899844da46f981c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:02.584565 21764 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Bootstrap starting.
I20260925 10:17:02.585130 21764 tablet_bootstrap.cc:654] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:02.585354 21764 log.cc:824] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:02.585767 21766 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Bootstrap starting.
I20260925 10:17:02.586345 21764 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: No bootstrap required, opened a new log
I20260925 10:17:02.586411 21764 ts_tablet_manager.cc:1397] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:17:02.586560 21767 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Bootstrap starting.
I20260925 10:17:02.587198 21767 tablet_bootstrap.cc:654] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:02.587550 21767 log.cc:824] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:02.588312 21767 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: No bootstrap required, opened a new log
I20260925 10:17:02.588389 21767 ts_tablet_manager.cc:1397] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:17:02.588568 21766 tablet_bootstrap.cc:654] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:02.588851 21766 log.cc:824] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:02.589272 21764 raft_consensus.cc:348] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.589437 21764 raft_consensus.cc:374] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:02.589471 21764 raft_consensus.cc:729] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9f68ca622a422c8e258bfd1a733680, State: Initialized, Role: FOLLOWER
I20260925 10:17:02.589622 21766 tablet_bootstrap.cc:492] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: No bootstrap required, opened a new log
I20260925 10:17:02.589594 21764 consensus_queue.cc:260] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.589690 21766 ts_tablet_manager.cc:1397] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 10:17:02.589861 21764 ts_tablet_manager.cc:1428] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:17:02.589985 21750 heartbeater.cc:503] Master 127.3.152.254:35787 was elected leader, sending a full tablet report...
I20260925 10:17:02.591101 21767 raft_consensus.cc:348] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.591217 21767 raft_consensus.cc:374] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:02.591238 21767 raft_consensus.cc:729] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a3b0678334410811710ca09f40648, State: Initialized, Role: FOLLOWER
I20260925 10:17:02.591351 21767 consensus_queue.cc:260] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.591610 21767 ts_tablet_manager.cc:1428] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:17:02.591655 21619 heartbeater.cc:503] Master 127.3.152.254:35787 was elected leader, sending a full tablet report...
I20260925 10:17:02.592033 21766 raft_consensus.cc:348] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.592212 21766 raft_consensus.cc:374] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:02.592257 21766 raft_consensus.cc:729] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498b426105854f329f125665bfa82d97, State: Initialized, Role: FOLLOWER
I20260925 10:17:02.592374 21766 consensus_queue.cc:260] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.592612 21766 ts_tablet_manager.cc:1428] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:17:02.592747 21488 heartbeater.cc:503] Master 127.3.152.254:35787 was elected leader, sending a full tablet report...
W20260925 10:17:02.652189 21620 tablet.cc:2369] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:17:02.776599 21489 tablet.cc:2369] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:17:02.798197 21772 raft_consensus.cc:482] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:02.798327 21772 raft_consensus.cc:504] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.798673 21772 leader_election.cc:288] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d9f68ca622a422c8e258bfd1a733680 (127.3.152.195:46219), 555a3b0678334410811710ca09f40648 (127.3.152.194:38719)
W20260925 10:17:02.802845 21751 tablet.cc:2369] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:17:02.802973 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.799056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:37709 (local address 127.3.152.194:38719)
0925 10:17:02.799105 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.799107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.799115 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:02.799185 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.799186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.799197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.799265 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:02.799495 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.800280 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.801963 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.802529 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.802607 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.802615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.802858 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.802903 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:17:02.803148 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.798900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38719 (local address 127.3.152.193:37709)
0925 10:17:02.799036 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.799048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.799126 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.799306 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.799307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.799308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:02.799415 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.799418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.800374 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.800376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.801851 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.801855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.802671 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.802672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.802764 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.802769 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.802839 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.802906 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":72,"thread_start_us":83,"threads_started":1}
I20260925 10:17:02.803208 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.798986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:43891 (local address 127.3.152.195:46219)
0925 10:17:02.799038 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.799041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.799147 (+ 106us) server_negotiation.cc:407] Connection header received
0925 10:17:02.799186 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.799187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.799196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.799270 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:02.799493 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.800281 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.801949 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.802529 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.802607 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.802615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.803103 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.803147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:17:02.803239 21773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.798902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:46219 (local address 127.3.152.193:43891)
0925 10:17:02.799075 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.799083 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:02.799136 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:02.799297 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:02.799298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:02.799300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:02.799415 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.799418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.800374 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.800376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.801852 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:02.801855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.802669 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.802671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:02.802764 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.802769 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:02.802839 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:17:02.802904 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":106,"thread_start_us":81,"threads_started":1}
I20260925 10:17:02.803552 21573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af03fc398c246d899844da46f981c25" candidate_uuid: "498b426105854f329f125665bfa82d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a3b0678334410811710ca09f40648" is_pre_election: true
I20260925 10:17:02.803614 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af03fc398c246d899844da46f981c25" candidate_uuid: "498b426105854f329f125665bfa82d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d9f68ca622a422c8e258bfd1a733680" is_pre_election: true
I20260925 10:17:02.803730 21704 raft_consensus.cc:2454] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498b426105854f329f125665bfa82d97 in term 0.
I20260925 10:17:02.803730 21573 raft_consensus.cc:2454] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498b426105854f329f125665bfa82d97 in term 0.
I20260925 10:17:02.804015 21376 leader_election.cc:302] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498b426105854f329f125665bfa82d97, 555a3b0678334410811710ca09f40648; no voters:
I20260925 10:17:02.804165 21772 raft_consensus.cc:2781] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:02.804244 21772 raft_consensus.cc:482] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:02.804263 21772 raft_consensus.cc:3037] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:02.805017 21772 raft_consensus.cc:504] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.805223 21772 leader_election.cc:288] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [CANDIDATE]: Term 1 election: Requested vote from peers 6d9f68ca622a422c8e258bfd1a733680 (127.3.152.195:46219), 555a3b0678334410811710ca09f40648 (127.3.152.194:38719)
I20260925 10:17:02.805356 21573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af03fc398c246d899844da46f981c25" candidate_uuid: "498b426105854f329f125665bfa82d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555a3b0678334410811710ca09f40648"
I20260925 10:17:02.805457 21704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af03fc398c246d899844da46f981c25" candidate_uuid: "498b426105854f329f125665bfa82d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d9f68ca622a422c8e258bfd1a733680"
I20260925 10:17:02.805451 21573 raft_consensus.cc:3037] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:02.805545 21704 raft_consensus.cc:3037] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:02.806239 21704 raft_consensus.cc:2454] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498b426105854f329f125665bfa82d97 in term 1.
I20260925 10:17:02.806280 21573 raft_consensus.cc:2454] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498b426105854f329f125665bfa82d97 in term 1.
I20260925 10:17:02.806473 21375 leader_election.cc:302] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498b426105854f329f125665bfa82d97, 6d9f68ca622a422c8e258bfd1a733680; no voters:
I20260925 10:17:02.806589 21772 raft_consensus.cc:2781] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:02.806721 21772 raft_consensus.cc:686] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 1 LEADER]: Becoming Leader. State: Replica: 498b426105854f329f125665bfa82d97, State: Running, Role: LEADER
I20260925 10:17:02.806844 21772 consensus_queue.cc:237] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } }
I20260925 10:17:02.807737 21299 catalog_manager.cc:5495] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 498b426105854f329f125665bfa82d97 (127.3.152.193). New cstate: current_term: 1 leader_uuid: "498b426105854f329f125665bfa82d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498b426105854f329f125665bfa82d97" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 } health_report { overall_health: UNKNOWN } } }
I20260925 10:17:02.864806 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.862187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.3.152.193:44261)
0925 10:17:02.862245 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:02.862247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:02.862258 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:02.862334 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:02.862334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:02.862356 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:02.862436 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:02.862653 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.863142 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.863943 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:02.864188 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:02.864269 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:02.864448 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:02.864566 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:02.864659 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:02.864683 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 10:17:02.864705 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:17:02.866755 21573 raft_consensus.cc:1261] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 1 FOLLOWER]: Refusing update from remote peer 498b426105854f329f125665bfa82d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:02.866756 21704 raft_consensus.cc:1261] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 1 FOLLOWER]: Refusing update from remote peer 498b426105854f329f125665bfa82d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:02.867123 21772 consensus_queue.cc:1035] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d9f68ca622a422c8e258bfd1a733680" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:02.867306 21775 consensus_queue.cc:1035] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555a3b0678334410811710ca09f40648" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:02.870497 21780 mvcc.cc:205] Tried to move back new op lower bound from 7333197508059357184 to 7333197507817607168. Current Snapshot: MvccSnapshot[applied={T|T < 7333197508059357184 or (T in {7333197508059357184})}]
I20260925 10:17:02.871984 21299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:02.872097 21299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:02.872882 21299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ea1115e71c146ceb1ba9cba2ae9ab6f: Sending DeleteTablet for 3 replicas of tablet 5af03fc398c246d899844da46f981c25
I20260925 10:17:02.873193 21422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af03fc398c246d899844da46f981c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:02 UTC) from {username='slave'} at 127.0.0.1:44694
I20260925 10:17:02.873235 21684 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af03fc398c246d899844da46f981c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:02 UTC) from {username='slave'} at 127.0.0.1:53680
I20260925 10:17:02.873265 21553 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af03fc398c246d899844da46f981c25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:02 UTC) from {username='slave'} at 127.0.0.1:53114
I20260925 10:17:02.873409 21788 tablet_replica.cc:330] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: stopping tablet replica
I20260925 10:17:02.873503 21788 raft_consensus.cc:2229] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:02.873586 21788 raft_consensus.cc:2258] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:02.873661 21789 tablet_replica.cc:330] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: stopping tablet replica
I20260925 10:17:02.873785 21789 raft_consensus.cc:2229] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:02.873907 21789 raft_consensus.cc:2258] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:02.873981 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21360
I20260925 10:17:02.874001 21788 ts_tablet_manager.cc:1905] T 5af03fc398c246d899844da46f981c25 P 498b426105854f329f125665bfa82d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:02.874352 21789 ts_tablet_manager.cc:1905] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:02.876300 21789 ts_tablet_manager.cc:1918] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:02.876380 21789 log.cc:1196] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/5af03fc398c246d899844da46f981c25
I20260925 10:17:02.876631 21789 ts_tablet_manager.cc:1939] T 5af03fc398c246d899844da46f981c25 P 6d9f68ca622a422c8e258bfd1a733680: Deleting consensus metadata
I20260925 10:17:02.876823 21790 tablet_replica.cc:330] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: stopping tablet replica
I20260925 10:17:02.876935 21790 raft_consensus.cc:2229] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:02.877043 21790 raft_consensus.cc:2258] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:02.877266 21284 catalog_manager.cc:4841] TS 6d9f68ca622a422c8e258bfd1a733680 (127.3.152.195:46219): tablet 5af03fc398c246d899844da46f981c25 (table test-table [id=706e7c809df443349707ce60c94bfaf2]) successfully deleted
I20260925 10:17:02.877550 21790 ts_tablet_manager.cc:1905] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:02.878988 21790 ts_tablet_manager.cc:1918] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:02.879050 21790 log.cc:1196] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/5af03fc398c246d899844da46f981c25
I20260925 10:17:02.879288 21790 ts_tablet_manager.cc:1939] T 5af03fc398c246d899844da46f981c25 P 555a3b0678334410811710ca09f40648: Deleting consensus metadata
I20260925 10:17:02.879683 21285 catalog_manager.cc:4841] TS 555a3b0678334410811710ca09f40648 (127.3.152.194:38719): tablet 5af03fc398c246d899844da46f981c25 (table test-table [id=706e7c809df443349707ce60c94bfaf2]) successfully deleted
W20260925 10:17:02.880709 21286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:44261 (error 108)
I20260925 10:17:02.880980 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21491
I20260925 10:17:02.881181 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:02.881046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44261 (local address 127.0.0.1:44726)
0925 10:17:02.881106 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:02.881145 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 10:17:02.882021 21286 catalog_manager.cc:4584] TS 498b426105854f329f125665bfa82d97 (127.3.152.193:44261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5af03fc398c246d899844da46f981c25: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44261: connect: Connection refused (error 111)
I20260925 10:17:02.885944 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21622
I20260925 10:17:02.890434 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21269
2026-09-25T10:17:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (951 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:02 dist-test-slave-dzwq krb5kdc[21794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:02 dist-test-slave-dzwq krb5kdc[21794](info): set up 2 sockets
Sep 25 10:17:02 dist-test-slave-dzwq krb5kdc[21794](info): commencing operation
krb5kdc: starting...
W20260925 10:17:03.928434 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:04.054958 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:39213
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:39213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:04.135852 21807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:04.136149 21807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:04.136186 21807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:04.137408 21807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:04.137485 21807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:04.137509 21807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:04.137532 21807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:04.137562 21807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:04.139067 21807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:39213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:39213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:04.139358 21807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:04.139626 21807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:04.144475 21812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.144466 21815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.144470 21813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:04.144762 21807 server_base.cc:1034] running on GCE node
I20260925 10:17:04.145017 21807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:04.145325 21807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:04.146469 21807 hybrid_clock.cc:648] HybridClock initialized: now 1790331424146444 us; error 39 us; skew 500 ppm
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:04.153556 21807 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:04.154052 21807 webserver.cc:460] Webserver started at http://127.3.152.254:41381/ using document root <none> and password file <none>
I20260925 10:17:04.154244 21807 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:04.154285 21807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:04.154376 21807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:04.155406 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "be95b6923ea79224229456e58817a6f9"
server_key_iv: "a4190002e12499542befc8f034b21a53"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.155725 21807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "be95b6923ea79224229456e58817a6f9"
server_key_iv: "a4190002e12499542befc8f034b21a53"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.157104 21807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:04.157881 21822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.158100 21807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:04.158195 21807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "be95b6923ea79224229456e58817a6f9"
server_key_iv: "a4190002e12499542befc8f034b21a53"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.158277 21807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:04.172263 21807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:04.174331 21807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:04.174479 21807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:04.178772 21807 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:39213
I20260925 10:17:04.178805 21874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:39213 every 8 connection(s)
I20260925 10:17:04.179176 21807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:04.180133 21875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:04.180152 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21807
I20260925 10:17:04.180284 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:04.180490 3683 external_mini_cluster.cc:1428] Setting key 94bf9cb8148db80e08be7ccfa23d8cd3
I20260925 10:17:04.183598 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: Bootstrap starting.
I20260925 10:17:04.184351 21875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:04.184646 21875 log.cc:824] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:04.185330 21875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: No bootstrap required, opened a new log
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:04.188167 21875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } }
I20260925 10:17:04.188347 21875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:04.188385 21875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f17a01d5f2e43e1b1d8e8936ae163e2, State: Initialized, Role: FOLLOWER
I20260925 10:17:04.188530 21875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } }
I20260925 10:17:04.188622 21875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:04.188658 21875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:04.188715 21875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:04.189437 21875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } }
I20260925 10:17:04.189688 21875 leader_election.cc:302] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f17a01d5f2e43e1b1d8e8936ae163e2; no voters:
I20260925 10:17:04.189882 21875 leader_election.cc:288] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:04.189944 21880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:04.190099 21880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0f17a01d5f2e43e1b1d8e8936ae163e2, State: Running, Role: LEADER
I20260925 10:17:04.190167 21875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:04.190207 21880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } }
I20260925 10:17:04.190522 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f17a01d5f2e43e1b1d8e8936ae163e2. Latest consensus state: current_term: 1 leader_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } } }
I20260925 10:17:04.190613 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:04.190528 21882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f17a01d5f2e43e1b1d8e8936ae163e2" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39213 } } }
I20260925 10:17:04.190748 21882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:04.191305 21886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:04.191737 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.181048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.3.152.254:39213)
0925 10:17:04.181230 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.181240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.181274 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:17:04.181825 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.181835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.181899 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.182051 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.182644 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.184096 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.184927 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.185354 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.185478 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.187790 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.187795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.187796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:04.187809 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.189903 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.190346 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.190351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.190351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.190449 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.190687 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.190688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.190688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.191025 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.191194 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.191341 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.191488 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260925 10:17:04.191862 21886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:04.193804 21886 catalog_manager.cc:1347] Generated new cluster ID: 6f8c4d9fc1cb4edcb42e671b6afea8cf
I20260925 10:17:04.193861 21886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:04.204663 21886 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:04.205188 21886 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:04.209877 21886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: Generated new TSK 0
I20260925 10:17:04.210069 21886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:17:04.243927 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:04.328800 21901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:04.329021 21901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:04.329051 21901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:04.330370 21901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:04.330420 21901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:04.330449 21901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:04.332134 21901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:04.332420 21901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:04.332698 21901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:04.338291 21906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.338300 21909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.338308 21907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:04.338613 21901 server_base.cc:1034] running on GCE node
I20260925 10:17:04.338842 21901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:04.339085 21901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:04.340238 21901 hybrid_clock.cc:648] HybridClock initialized: now 1790331424340228 us; error 36 us; skew 500 ppm
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:04.342916 21901 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:17:04.343421 21901 webserver.cc:460] Webserver started at http://127.3.152.193:44321/ using document root <none> and password file <none>
I20260925 10:17:04.343679 21901 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:04.343753 21901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:04.343902 21901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:04.345093 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "0421338ce64c40b9b42a680aa9434751"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "f86f67121fad81bc327fbfb0221ba4fe"
server_key_iv: "23da9c9f5bf6b0f5f9091312f0485515"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.345484 21901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "0421338ce64c40b9b42a680aa9434751"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "f86f67121fad81bc327fbfb0221ba4fe"
server_key_iv: "23da9c9f5bf6b0f5f9091312f0485515"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.346889 21901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:04.347672 21916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.347864 21901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:04.347999 21901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "0421338ce64c40b9b42a680aa9434751"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "f86f67121fad81bc327fbfb0221ba4fe"
server_key_iv: "23da9c9f5bf6b0f5f9091312f0485515"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.348104 21901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:04.363617 21901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:04.365821 21901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:04.365995 21901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:04.366242 21901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:04.366570 21901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:04.366627 21901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.366670 21901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:04.366694 21901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.372505 21901 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:44159
I20260925 10:17:04.372556 22029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:44159 every 8 connection(s)
I20260925 10:17:04.372884 21901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:04.379284 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 21901
I20260925 10:17:04.379427 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:04.379680 3683 external_mini_cluster.cc:1428] Setting key d2454d383587ab961855959a08318ed4
I20260925 10:17:04.382577 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.373576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:34713 (local address 127.3.152.254:39213)
0925 10:17:04.373644 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.373646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.373866 (+ 220us) server_negotiation.cc:407] Connection header received
0925 10:17:04.374406 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.374407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.374419 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.374477 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.375463 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.376068 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.377086 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.377326 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.377406 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.379631 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.379633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.379637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:04.379644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.381141 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.381569 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.381570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.381571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.381633 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.381906 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.381907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.381907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.382026 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.382096 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.382390 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.382486 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:17:04.382778 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.373553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39213 (local address 127.3.152.193:34713)
0925 10:17:04.373735 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.373773 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.374296 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.374558 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.374562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.375139 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:04.375343 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.375348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.376164 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.376165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.376979 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.376981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.377414 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.377415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.377538 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.378082 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:04.378091 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:04.379518 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:04.381250 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.381258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.381260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.381489 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.381714 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.381715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.381715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.381831 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.382098 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:04.382106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:04.382252 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.382357 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.382521 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260925 10:17:04.383256 22030 heartbeater.cc:347] Connected to a master server at 127.3.152.254:39213
I20260925 10:17:04.383349 22030 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:04.383540 22030 heartbeater.cc:511] Master 127.3.152.254:39213 requested a full tablet report, sending...
I20260925 10:17:04.384163 21839 ts_manager.cc:194] Registered new tserver with Master: 0421338ce64c40b9b42a680aa9434751 (127.3.152.193:44159)
I20260925 10:17:04.384783 21839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:34713
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:17:04.406719 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:04.492273 22035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:04.492484 22035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:04.492509 22035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:04.493634 22035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:04.493673 22035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:04.493700 22035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:04.495111 22035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:04.495311 22035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:04.495500 22035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:04.500684 22040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.500701 22041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.500692 22043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:04.500892 22035 server_base.cc:1034] running on GCE node
I20260925 10:17:04.501093 22035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:04.501308 22035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:04.502467 22035 hybrid_clock.cc:648] HybridClock initialized: now 1790331424502447 us; error 32 us; skew 500 ppm
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:04.505292 22035 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:17:04.505899 22035 webserver.cc:460] Webserver started at http://127.3.152.194:45429/ using document root <none> and password file <none>
I20260925 10:17:04.506155 22035 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:04.506206 22035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:04.506384 22035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:04.507525 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "c052a39116d44723b998e4341b199645"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "4b101c48587a1d7d91a330ae33c3e315"
server_key_iv: "99a9904182a713a6230a656185ca2d29"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.507923 22035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "c052a39116d44723b998e4341b199645"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "4b101c48587a1d7d91a330ae33c3e315"
server_key_iv: "99a9904182a713a6230a656185ca2d29"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.509379 22035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:17:04.510267 22050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.510448 22035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:04.510637 22035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "c052a39116d44723b998e4341b199645"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "4b101c48587a1d7d91a330ae33c3e315"
server_key_iv: "99a9904182a713a6230a656185ca2d29"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.510726 22035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:04.531816 22035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:04.533864 22035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:04.534057 22035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:04.534325 22035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:04.534720 22035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:04.534788 22035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.534837 22035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:04.534871 22035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.541430 22035 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:37379
I20260925 10:17:04.541481 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:37379 every 8 connection(s)
I20260925 10:17:04.541822 22035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:04.542526 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22035
I20260925 10:17:04.542666 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:04.542919 3683 external_mini_cluster.cc:1428] Setting key 613a366272503757bb891a8419e9c93f
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:17:04.552026 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.542587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:45579 (local address 127.3.152.254:39213)
0925 10:17:04.542696 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.542698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.543027 (+ 329us) server_negotiation.cc:407] Connection header received
0925 10:17:04.543719 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.543720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.543731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.543792 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.544830 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.545328 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.546357 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.546551 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.546626 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.548897 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.548898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.548899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:04.548906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.550337 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.550847 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.550848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.550848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.550908 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.551230 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.551231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.551232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.551354 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.551431 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.551747 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.551844 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 10:17:04.552248 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.542592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39213 (local address 127.3.152.194:45579)
0925 10:17:04.542875 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.542924 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.543587 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.543866 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.543870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.544526 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:04.544728 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.544733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.545428 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.545428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.546252 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.546254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.546639 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.546640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.546779 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.547331 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:04.547339 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:04.548788 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:04.550437 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.550444 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.550447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.550775 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.550981 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.550982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.550982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.551137 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.551422 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:04.551429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:04.551574 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.551687 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.552011 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":102,"threads_started":1}
I20260925 10:17:04.552691 22164 heartbeater.cc:347] Connected to a master server at 127.3.152.254:39213
I20260925 10:17:04.552814 22164 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:04.552989 22164 heartbeater.cc:511] Master 127.3.152.254:39213 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
I20260925 10:17:04.553390 21839 ts_manager.cc:194] Registered new tserver with Master: c052a39116d44723b998e4341b199645 (127.3.152.194:37379)
I20260925 10:17:04.553915 21839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:45579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:17:04.570911 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:04.653105 22169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:04.653337 22169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:04.653378 22169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:04.654543 22169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:04.654615 22169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:04.654663 22169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:04.656149 22169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:39213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:04.656424 22169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:04.656662 22169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:04.661809 22174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.661801 22175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.662067 22177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:04.662189 22169 server_base.cc:1034] running on GCE node
I20260925 10:17:04.662393 22169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:04.662645 22169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:04.663836 22169 hybrid_clock.cc:648] HybridClock initialized: now 1790331424663824 us; error 31 us; skew 500 ppm
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:04.666363 22169 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:17:04.666882 22169 webserver.cc:460] Webserver started at http://127.3.152.195:45017/ using document root <none> and password file <none>
I20260925 10:17:04.667112 22169 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:04.667181 22169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:04.667330 22169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:04.668535 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "d04937c66bfa45b6b2227a34ec2ac249"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "c02db5e87559da37946d4c476992a5c0"
server_key_iv: "850d181f64616e9f7a5075d4827980c8"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.668896 22169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "d04937c66bfa45b6b2227a34ec2ac249"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "c02db5e87559da37946d4c476992a5c0"
server_key_iv: "850d181f64616e9f7a5075d4827980c8"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.670217 22169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:04.670955 22184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.671125 22169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:04.671228 22169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "d04937c66bfa45b6b2227a34ec2ac249"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "c02db5e87559da37946d4c476992a5c0"
server_key_iv: "850d181f64616e9f7a5075d4827980c8"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.671300 22169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:04.680693 22169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:04.682510 22169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:04.682673 22169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:04.682896 22169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:04.683198 22169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:04.683252 22169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.683290 22169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:04.683319 22169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.689184 22169 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:33105
I20260925 10:17:04.689227 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:33105 every 8 connection(s)
I20260925 10:17:04.689540 22169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
Sep 25 10:17:04 dist-test-slave-dzwq krb5kdc[21794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:04.696019 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22169
I20260925 10:17:04.696111 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:04.696316 3683 external_mini_cluster.cc:1428] Setting key ea079fc25f73f01dbe47666d43b88fea
I20260925 10:17:04.699280 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.690241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:36977 (local address 127.3.152.254:39213)
0925 10:17:04.690308 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.690311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.690487 (+ 176us) server_negotiation.cc:407] Connection header received
0925 10:17:04.691152 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.691153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.691166 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.691232 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.692336 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.692857 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.693864 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.694092 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.694180 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.696364 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.696365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.696365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:04.696373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.698142 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.698492 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.698493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.698493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.698544 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.698754 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.698755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.698755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.698855 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.698905 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.699155 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.699217 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:17:04.699448 22300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.690195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39213 (local address 127.3.152.195:36977)
0925 10:17:04.690368 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.690409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.691049 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.691301 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.691305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.692029 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:04.692238 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.692244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.692968 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.692969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.693769 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.693771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.694192 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.694193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.694315 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.694771 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:04.694779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:04.696251 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:04.698212 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.698218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.698219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.698427 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.698602 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:04.698602 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:04.698602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:04.698700 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:04.698915 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:04.698920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:04.699046 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.699131 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.699251 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":88,"threads_started":1}
I20260925 10:17:04.699952 22298 heartbeater.cc:347] Connected to a master server at 127.3.152.254:39213
I20260925 10:17:04.700073 22298 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:04.700248 22298 heartbeater.cc:511] Master 127.3.152.254:39213 requested a full tablet report, sending...
I20260925 10:17:04.700665 21839 ts_manager.cc:194] Registered new tserver with Master: d04937c66bfa45b6b2227a34ec2ac249 (127.3.152.195:33105)
I20260925 10:17:04.701009 21839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:36977
I20260925 10:17:04.708684 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:04.716519 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.710111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.3.152.254:39213)
0925 10:17:04.710187 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.710189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.710224 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:17:04.710317 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.710317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.710326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.710380 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.710960 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.711439 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.712286 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.712473 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.712520 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.713900 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.713900 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.713901 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:04.713907 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.715352 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.715716 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.715717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.715717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.715768 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:04.716002 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:04.716003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:04.716004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:04.716144 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.716227 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.716355 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.716404 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:17:04.718040 21839 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:04.719578 21839 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:04.726020 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.722261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.3.152.193:44159)
0925 10:17:04.722439 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.722444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.722454 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:04.722491 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.722493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.722508 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.722588 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:04.722804 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.723873 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725072 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.725564 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725666 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.725732 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.725884 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.725927 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260925 10:17:04.726006 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.722043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44159 (local address 127.0.0.1:46138)
0925 10:17:04.722188 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.722199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.722244 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.722671 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.722672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.722674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:04.722725 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.722727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.724000 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.724001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.724952 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.724955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.725668 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.725747 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.725762 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.725862 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.725922 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260925 10:17:04.726248 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.721851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:37379 (local address 127.0.0.1:49052)
0925 10:17:04.721999 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.722040 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.722109 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.722632 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.722633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.722635 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:04.722692 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.722694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.723658 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.723660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.724965 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.724967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.725992 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.726079 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.726082 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.726115 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.726164 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20260925 10:17:04.726337 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.721944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.3.152.195:33105)
0925 10:17:04.722269 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.722274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.722286 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:17:04.722328 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.722330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.722346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.722422 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:04.722652 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.723798 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725155 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.725894 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725958 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.726012 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.726207 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.726246 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":80,"threads_started":1}
I20260925 10:17:04.726480 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.721863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:33105 (local address 127.0.0.1:48624)
0925 10:17:04.722141 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.722152 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.722203 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.722480 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.722484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.722486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:04.722566 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.722568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.723901 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.723902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.724990 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.724992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.726042 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.726043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.726093 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.726097 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.726143 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.726171 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20260925 10:17:04.726652 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.721996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.3.152.194:37379)
0925 10:17:04.722354 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.722357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.722368 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:04.722403 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.722405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.722419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.722787 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:04.722853 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.723509 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.725111 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.725875 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.726429 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.726496 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.726523 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.726572 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":107,"threads_started":1}
I20260925 10:17:04.726974 21964 tablet_service.cc:1467] Processing CreateTablet for tablet f20f14df17d04c99888e62e0ae951363 (DEFAULT_TABLE table=test-table [id=18ca0a13e75946078f3e65d7a28345e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:04.727368 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20f14df17d04c99888e62e0ae951363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:04.728420 22098 tablet_service.cc:1467] Processing CreateTablet for tablet f20f14df17d04c99888e62e0ae951363 (DEFAULT_TABLE table=test-table [id=18ca0a13e75946078f3e65d7a28345e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:04.728760 22098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20f14df17d04c99888e62e0ae951363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:04.730772 22315 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: Bootstrap starting.
I20260925 10:17:04.731382 22315 tablet_bootstrap.cc:654] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:04.731818 22316 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Bootstrap starting.
I20260925 10:17:04.731958 22315 log.cc:824] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:04.732596 22316 tablet_bootstrap.cc:654] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:04.732576 22232 tablet_service.cc:1467] Processing CreateTablet for tablet f20f14df17d04c99888e62e0ae951363 (DEFAULT_TABLE table=test-table [id=18ca0a13e75946078f3e65d7a28345e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:04.732897 22232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20f14df17d04c99888e62e0ae951363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:04.733083 22315 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: No bootstrap required, opened a new log
I20260925 10:17:04.733146 22315 ts_tablet_manager.cc:1397] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:17:04.733198 22316 log.cc:824] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:04.733870 22316 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: No bootstrap required, opened a new log
I20260925 10:17:04.733954 22316 ts_tablet_manager.cc:1397] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:04.735146 22315 raft_consensus.cc:348] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.735332 22315 raft_consensus.cc:374] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:04.735365 22315 raft_consensus.cc:729] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0421338ce64c40b9b42a680aa9434751, State: Initialized, Role: FOLLOWER
I20260925 10:17:04.735488 22315 consensus_queue.cc:260] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.735424 22316 raft_consensus.cc:348] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.735633 22316 raft_consensus.cc:374] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:04.735677 22316 raft_consensus.cc:729] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c052a39116d44723b998e4341b199645, State: Initialized, Role: FOLLOWER
I20260925 10:17:04.735788 22315 ts_tablet_manager.cc:1428] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:17:04.735798 22316 consensus_queue.cc:260] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.735924 22030 heartbeater.cc:503] Master 127.3.152.254:39213 was elected leader, sending a full tablet report...
I20260925 10:17:04.736099 22316 ts_tablet_manager.cc:1428] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 10:17:04.736259 22164 heartbeater.cc:503] Master 127.3.152.254:39213 was elected leader, sending a full tablet report...
I20260925 10:17:04.736755 22321 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Bootstrap starting.
I20260925 10:17:04.737311 22321 tablet_bootstrap.cc:654] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:04.737905 22321 log.cc:824] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:04.738528 22321 tablet_bootstrap.cc:492] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: No bootstrap required, opened a new log
I20260925 10:17:04.738641 22321 ts_tablet_manager.cc:1397] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:17:04.739898 22321 raft_consensus.cc:348] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.740103 22321 raft_consensus.cc:374] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:04.740131 22321 raft_consensus.cc:729] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04937c66bfa45b6b2227a34ec2ac249, State: Initialized, Role: FOLLOWER
I20260925 10:17:04.740217 22321 consensus_queue.cc:260] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.740476 22321 ts_tablet_manager.cc:1428] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:04.740504 22298 heartbeater.cc:503] Master 127.3.152.254:39213 was elected leader, sending a full tablet report...
I20260925 10:17:04.746119 22320 raft_consensus.cc:482] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:04.746215 22320 raft_consensus.cc:504] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.746498 22320 leader_election.cc:288] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d04937c66bfa45b6b2227a34ec2ac249 (127.3.152.195:33105), 0421338ce64c40b9b42a680aa9434751 (127.3.152.193:44159)
I20260925 10:17:04.749420 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.746654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:33105 (local address 127.3.152.194:53941)
0925 10:17:04.746755 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.746767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.746818 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.746946 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.746947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.746949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:04.747018 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.747019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.747735 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.747736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.748582 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.748584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.749135 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.749136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.749234 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.749239 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.749289 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.749335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 10:17:04.749498 22311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.746701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:53941 (local address 127.3.152.195:33105)
0925 10:17:04.746760 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.746762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.746817 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:17:04.746864 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.746865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.746876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.746935 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:04.747116 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.747648 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.748666 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.749053 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.749114 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.749121 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.749387 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.749436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:17:04.749555 22314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.746973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:48301 (local address 127.3.152.193:44159)
0925 10:17:04.747028 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.747030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.747038 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:04.747063 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.747064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.747073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.747124 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:04.747272 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.747757 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.748744 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.749163 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.749248 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.749257 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.749439 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.749483 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 10:17:04.749804 22324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.746773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44159 (local address 127.3.152.194:48301)
0925 10:17:04.746952 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.746962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:04.747006 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:04.747134 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:04.747135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:04.747137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:04.747207 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.747208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.747835 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.747836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.748649 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:04.748652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.749272 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.749273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:04.749347 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.749351 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:04.749648 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 10:17:04.749699 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260925 10:17:04.749778 22252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20f14df17d04c99888e62e0ae951363" candidate_uuid: "c052a39116d44723b998e4341b199645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04937c66bfa45b6b2227a34ec2ac249" is_pre_election: true
I20260925 10:17:04.749925 22252 raft_consensus.cc:2454] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c052a39116d44723b998e4341b199645 in term 0.
I20260925 10:17:04.750123 22051 leader_election.cc:302] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c052a39116d44723b998e4341b199645, d04937c66bfa45b6b2227a34ec2ac249; no voters:
I20260925 10:17:04.750128 21984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20f14df17d04c99888e62e0ae951363" candidate_uuid: "c052a39116d44723b998e4341b199645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0421338ce64c40b9b42a680aa9434751" is_pre_election: true
I20260925 10:17:04.750255 22320 raft_consensus.cc:2781] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:04.750269 21984 raft_consensus.cc:2454] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c052a39116d44723b998e4341b199645 in term 0.
I20260925 10:17:04.750311 22320 raft_consensus.cc:482] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:04.750334 22320 raft_consensus.cc:3037] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:04.751156 22320 raft_consensus.cc:504] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.751302 22320 leader_election.cc:288] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [CANDIDATE]: Term 1 election: Requested vote from peers d04937c66bfa45b6b2227a34ec2ac249 (127.3.152.195:33105), 0421338ce64c40b9b42a680aa9434751 (127.3.152.193:44159)
I20260925 10:17:04.751549 21984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20f14df17d04c99888e62e0ae951363" candidate_uuid: "c052a39116d44723b998e4341b199645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0421338ce64c40b9b42a680aa9434751"
I20260925 10:17:04.751521 22252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20f14df17d04c99888e62e0ae951363" candidate_uuid: "c052a39116d44723b998e4341b199645" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04937c66bfa45b6b2227a34ec2ac249"
I20260925 10:17:04.751624 21984 raft_consensus.cc:3037] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:04.751626 22252 raft_consensus.cc:3037] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:04.752606 21984 raft_consensus.cc:2454] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c052a39116d44723b998e4341b199645 in term 1.
I20260925 10:17:04.752606 22252 raft_consensus.cc:2454] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c052a39116d44723b998e4341b199645 in term 1.
I20260925 10:17:04.752779 22051 leader_election.cc:302] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0421338ce64c40b9b42a680aa9434751, c052a39116d44723b998e4341b199645; no voters:
I20260925 10:17:04.752905 22320 raft_consensus.cc:2781] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:04.753091 22320 raft_consensus.cc:686] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 1 LEADER]: Becoming Leader. State: Replica: c052a39116d44723b998e4341b199645, State: Running, Role: LEADER
I20260925 10:17:04.753211 22320 consensus_queue.cc:237] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } }
I20260925 10:17:04.754117 21838 catalog_manager.cc:5495] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 reported cstate change: term changed from 0 to 1, leader changed from <none> to c052a39116d44723b998e4341b199645 (127.3.152.194). New cstate: current_term: 1 leader_uuid: "c052a39116d44723b998e4341b199645" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c052a39116d44723b998e4341b199645" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 37379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 } health_report { overall_health: UNKNOWN } } }
I20260925 10:17:04.761633 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.759269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.3.152.194:37379)
0925 10:17:04.759327 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.759329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.759337 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:04.759382 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.759382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.759398 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.759455 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:04.759583 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.760124 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.760877 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.761080 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.761155 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.761310 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:04.761408 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:04.761516 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.761547 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.761558 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:17:04.763439 21984 raft_consensus.cc:1261] T f20f14df17d04c99888e62e0ae951363 P 0421338ce64c40b9b42a680aa9434751 [term 1 FOLLOWER]: Refusing update from remote peer c052a39116d44723b998e4341b199645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:04.763439 22252 raft_consensus.cc:1261] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 1 FOLLOWER]: Refusing update from remote peer c052a39116d44723b998e4341b199645: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:04.763818 22320 consensus_queue.cc:1035] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0421338ce64c40b9b42a680aa9434751" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 44159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:04.763957 22325 consensus_queue.cc:1035] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04937c66bfa45b6b2227a34ec2ac249" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:04.766503 22328 mvcc.cc:205] Tried to move back new op lower bound from 7333197515828166656 to 7333197515789963264. Current Snapshot: MvccSnapshot[applied={T|T < 7333197515828166656}]
I20260925 10:17:04.766664 22329 mvcc.cc:205] Tried to move back new op lower bound from 7333197515828166656 to 7333197515789963264. Current Snapshot: MvccSnapshot[applied={T|T < 7333197515828166656}]
I20260925 10:17:04.768900 21839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:04.769001 21839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:04.769755 21839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f17a01d5f2e43e1b1d8e8936ae163e2: Sending DeleteTablet for 3 replicas of tablet f20f14df17d04c99888e62e0ae951363
I20260925 10:17:04.770066 22098 tablet_service.cc:1514] Processing DeleteTablet for tablet f20f14df17d04c99888e62e0ae951363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:04 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:49052
I20260925 10:17:04.770112 21964 tablet_service.cc:1514] Processing DeleteTablet for tablet f20f14df17d04c99888e62e0ae951363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:04 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:46138
I20260925 10:17:04.770284 22232 tablet_service.cc:1514] Processing DeleteTablet for tablet f20f14df17d04c99888e62e0ae951363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:04 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:48624
I20260925 10:17:04.770371 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21901
I20260925 10:17:04.770478 22337 tablet_replica.cc:330] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: stopping tablet replica
I20260925 10:17:04.770663 22339 tablet_replica.cc:330] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: stopping tablet replica
I20260925 10:17:04.770746 22339 raft_consensus.cc:2229] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:04.770929 22339 raft_consensus.cc:2258] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:04.771097 22337 raft_consensus.cc:2229] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:04.771185 22337 raft_consensus.cc:2258] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:04.771649 22339 ts_tablet_manager.cc:1905] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:04.773612 22337 ts_tablet_manager.cc:1905] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:04.775235 22337 ts_tablet_manager.cc:1918] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:04.775313 22337 log.cc:1196] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/f20f14df17d04c99888e62e0ae951363
I20260925 10:17:04.775614 22337 ts_tablet_manager.cc:1939] T f20f14df17d04c99888e62e0ae951363 P c052a39116d44723b998e4341b199645: Deleting consensus metadata
I20260925 10:17:04.776108 21826 catalog_manager.cc:4841] TS c052a39116d44723b998e4341b199645 (127.3.152.194:37379): tablet f20f14df17d04c99888e62e0ae951363 (table test-table [id=18ca0a13e75946078f3e65d7a28345e3]) successfully deleted
W20260925 10:17:04.776891 21823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:44159 (error 108)
I20260925 10:17:04.777123 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22035
I20260925 10:17:04.777794 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.777258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:44159 (local address 127.0.0.1:46142)
0925 10:17:04.777727 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:04.777765 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451}
I20260925 10:17:04.777810 22339 ts_tablet_manager.cc:1918] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:04.777884 22339 log.cc:1196] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/f20f14df17d04c99888e62e0ae951363
W20260925 10:17:04.777961 21823 catalog_manager.cc:4584] TS 0421338ce64c40b9b42a680aa9434751 (127.3.152.193:44159): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f20f14df17d04c99888e62e0ae951363: Network error: Client connection negotiation failed: client connection to 127.3.152.193:44159: connect: Connection refused (error 111)
I20260925 10:17:04.778249 22339 ts_tablet_manager.cc:1939] T f20f14df17d04c99888e62e0ae951363 P d04937c66bfa45b6b2227a34ec2ac249: Deleting consensus metadata
I20260925 10:17:04.778668 21823 catalog_manager.cc:4841] TS d04937c66bfa45b6b2227a34ec2ac249 (127.3.152.195:33105): tablet f20f14df17d04c99888e62e0ae951363 (table test-table [id=18ca0a13e75946078f3e65d7a28345e3]) successfully deleted
I20260925 10:17:04.783351 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22169
I20260925 10:17:04.787946 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 21807
2026-09-25T10:17:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1898 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T10:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:04Z Disabled control of system clock
I20260925 10:17:04.815749 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:44725
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:44725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:04.895780 22344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:04.896008 22344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:04.896031 22344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:04.897121 22344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:04.897156 22344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:04.897168 22344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:04.897181 22344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:04.897192 22344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:04.898447 22344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:44725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:44725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:04.898631 22344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:04.898862 22344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:04.903628 22349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.903630 22350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:04.903641 22352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:04.903872 22344 server_base.cc:1034] running on GCE node
I20260925 10:17:04.904156 22344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:04.904410 22344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:04.905551 22344 hybrid_clock.cc:648] HybridClock initialized: now 1790331424905527 us; error 39 us; skew 500 ppm
I20260925 10:17:04.906896 22344 webserver.cc:460] Webserver started at http://127.3.152.254:43535/ using document root <none> and password file <none>
I20260925 10:17:04.907162 22344 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:04.907217 22344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:04.907402 22344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:04.908540 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0a1dc2592fc0429eb3643c770fe9debc"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "689acd44e89e0dcd05e501a9e8889c85"
server_key_iv: "d85a312af757540826e74320a1fb2a5d"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.908911 22344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0a1dc2592fc0429eb3643c770fe9debc"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "689acd44e89e0dcd05e501a9e8889c85"
server_key_iv: "d85a312af757540826e74320a1fb2a5d"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.910276 22344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:04.911038 22358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:04.911239 22344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:04.911304 22344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0a1dc2592fc0429eb3643c770fe9debc"
format_stamp: "Formatted at 2026-09-25 10:17:04 on dist-test-slave-dzwq"
server_key: "689acd44e89e0dcd05e501a9e8889c85"
server_key_iv: "d85a312af757540826e74320a1fb2a5d"
server_key_version: "encryptionkey@0"
I20260925 10:17:04.911396 22344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:04.933868 22344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:04.934237 22344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:04.934384 22344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:04.938608 22344 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:44725
I20260925 10:17:04.938695 22410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:44725 every 8 connection(s)
I20260925 10:17:04.939034 22344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:04.940171 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22344
I20260925 10:17:04.940265 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:04.940207 22411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:04.940493 3683 external_mini_cluster.cc:1428] Setting key 42b0e76ec2b427e72fcf2b83c2a2b6af
I20260925 10:17:04.944363 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: Bootstrap starting.
I20260925 10:17:04.945137 22411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:04.945509 22411 log.cc:824] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:04.946287 22411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: No bootstrap required, opened a new log
I20260925 10:17:04.947526 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:04.941089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38502 (local address 127.3.152.254:44725)
0925 10:17:04.941319 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:04.941329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:04.941368 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:17:04.942196 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:04.942206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:04.942300 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:04.942483 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:04.942629 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.944215 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.945061 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:04.945761 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:04.945940 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:04.946790 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:04.946795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:04.946795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:04.947005 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:04.947041 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:04.947053 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:04.947137 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:04.947242 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:17:04.947335 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260925 10:17:04.948438 22411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } }
I20260925 10:17:04.948602 22411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:04.948626 22411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a1dc2592fc0429eb3643c770fe9debc, State: Initialized, Role: FOLLOWER
I20260925 10:17:04.948750 22411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } }
I20260925 10:17:04.948824 22411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:04.948848 22411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:04.948895 22411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:04.949442 22411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } }
I20260925 10:17:04.949556 22411 leader_election.cc:302] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a1dc2592fc0429eb3643c770fe9debc; no voters:
I20260925 10:17:04.949745 22411 leader_election.cc:288] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:04.949884 22416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:04.950014 22411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:04.950088 22416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [term 1 LEADER]: Becoming Leader. State: Replica: 0a1dc2592fc0429eb3643c770fe9debc, State: Running, Role: LEADER
I20260925 10:17:04.950204 22416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } }
I20260925 10:17:04.950733 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a1dc2592fc0429eb3643c770fe9debc. Latest consensus state: current_term: 1 leader_uuid: "0a1dc2592fc0429eb3643c770fe9debc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } } }
I20260925 10:17:04.950855 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:04.950745 22417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a1dc2592fc0429eb3643c770fe9debc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a1dc2592fc0429eb3643c770fe9debc" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 44725 } } }
I20260925 10:17:04.951339 22417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:04.951481 22425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:04.952239 22425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:04.954077 22425 catalog_manager.cc:1347] Generated new cluster ID: 76524eba9bf7467082fe9ae34a8ecf0f
I20260925 10:17:04.954140 22425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:04.963085 22425 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:04.963729 22425 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:04.969533 22425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: Generated new TSK 0
I20260925 10:17:04.969748 22425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:04.975621 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--builtin_ntp_servers=127.3.152.212:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:05.058715 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:05.058966 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:05.059006 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:05.060197 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:05.060266 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:05.060308 22435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:05.061709 22435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:05.061993 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:05.062346 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:05.068329 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:05.068344 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:05.068337 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:05.068667 22435 server_base.cc:1034] running on GCE node
I20260925 10:17:05.068866 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:05.069096 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:05.070289 22435 hybrid_clock.cc:648] HybridClock initialized: now 1790331425070277 us; error 33 us; skew 500 ppm
I20260925 10:17:05.071787 22435 webserver.cc:460] Webserver started at http://127.3.152.193:44667/ using document root <none> and password file <none>
I20260925 10:17:05.072103 22435 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:05.072175 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:05.072315 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:05.073403 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "0ee5c7a9cf454196951f8f21c1db7a69"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "ca6ccf507dd850162edae790d2cc37b8"
server_key_iv: "8cfa52641712396825eeaf72cfa28152"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.073791 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "0ee5c7a9cf454196951f8f21c1db7a69"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "ca6ccf507dd850162edae790d2cc37b8"
server_key_iv: "8cfa52641712396825eeaf72cfa28152"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.075157 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:17:05.075995 22449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.076210 22435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:17:05.076308 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "0ee5c7a9cf454196951f8f21c1db7a69"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "ca6ccf507dd850162edae790d2cc37b8"
server_key_iv: "8cfa52641712396825eeaf72cfa28152"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.076399 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:05.097400 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:05.097762 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:05.097920 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:05.098173 22435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:05.098511 22435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:05.098645 22435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.098693 22435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:05.098718 22435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.104694 22435 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34183
I20260925 10:17:05.104739 22562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34183 every 8 connection(s)
I20260925 10:17:05.105086 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:17:05.110378 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.105773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:51987 (local address 127.3.152.254:44725)
0925 10:17:05.105857 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.105860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.106113 (+ 253us) server_negotiation.cc:407] Connection header received
0925 10:17:05.106602 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.106603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.106617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.106672 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:05.107349 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.107859 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.108939 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.109143 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.109203 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.110123 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:05.110124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:05.110125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:05.110129 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:05.110142 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:05.110146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:05.110192 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.110286 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.110313 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:17:05.110647 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.105728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44725 (local address 127.3.152.193:51987)
0925 10:17:05.105938 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.105996 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.106517 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.106731 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.106734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.106743 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:05.107261 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.107265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.108006 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.108007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.108856 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.108858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.109252 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.109253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.109373 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.109986 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:05.109987 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:05.110016 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:05.110039 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:05.110214 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:05.110218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:05.110221 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.110275 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.110361 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":109,"threads_started":1}
I20260925 10:17:05.111260 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22435
I20260925 10:17:05.111300 22563 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44725
I20260925 10:17:05.111357 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:05.111423 22563 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:05.111615 22563 heartbeater.cc:511] Master 127.3.152.254:44725 requested a full tablet report, sending...
I20260925 10:17:05.111619 3683 external_mini_cluster.cc:1428] Setting key e046e57a57f27a3c04f0cdbaf8e61d92
I20260925 10:17:05.112202 22375 ts_manager.cc:194] Registered new tserver with Master: 0ee5c7a9cf454196951f8f21c1db7a69 (127.3.152.193:34183)
I20260925 10:17:05.112718 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--builtin_ntp_servers=127.3.152.212:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 10:17:05.112948 22375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.193:51987
W20260925 10:17:05.193603 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:05.193849 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:05.193884 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:05.195029 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:05.195093 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:05.195134 22566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:05.196542 22566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:05.196815 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:05.197113 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:05.202471 22572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:05.202471 22574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:05.202533 22566 server_base.cc:1034] running on GCE node
W20260925 10:17:05.202505 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:05.202914 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:05.203168 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:05.204334 22566 hybrid_clock.cc:648] HybridClock initialized: now 1790331425204338 us; error 46 us; skew 500 ppm
I20260925 10:17:05.205648 22566 webserver.cc:460] Webserver started at http://127.3.152.194:43361/ using document root <none> and password file <none>
I20260925 10:17:05.205907 22566 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:05.205981 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:05.206143 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:05.207301 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "c74733bee21a49ca8251186f3af97ec8"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "68ccf87ef826929d9a226237a26f8079"
server_key_iv: "fdfae884bbfcc5dd5274c503f05dcc6c"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.207684 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "c74733bee21a49ca8251186f3af97ec8"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "68ccf87ef826929d9a226237a26f8079"
server_key_iv: "fdfae884bbfcc5dd5274c503f05dcc6c"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.209097 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:05.209947 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.210180 22566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:05.210289 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "c74733bee21a49ca8251186f3af97ec8"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "68ccf87ef826929d9a226237a26f8079"
server_key_iv: "fdfae884bbfcc5dd5274c503f05dcc6c"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.210374 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:05.220065 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:05.220383 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:05.220534 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:05.220763 22566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:05.221099 22566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:05.221156 22566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.221205 22566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:05.221253 22566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.227079 22566 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:35165
I20260925 10:17:05.227197 22693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:35165 every 8 connection(s)
I20260925 10:17:05.227553 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:05.232753 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.228339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:34931 (local address 127.3.152.254:44725)
0925 10:17:05.228405 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.228407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.228685 (+ 278us) server_negotiation.cc:407] Connection header received
0925 10:17:05.229155 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.229156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.229167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.229223 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:05.229868 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.230351 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.231330 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.231506 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.231554 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.232464 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:05.232466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:05.232466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:05.232471 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:05.232485 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:05.232489 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:05.232533 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.232664 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.232688 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 10:17:05.233000 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.228334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44725 (local address 127.3.152.194:34931)
0925 10:17:05.228516 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.228585 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.229070 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.229277 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.229280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.229288 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:05.229793 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.229798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.230431 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.230432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.231251 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.231254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.231605 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.231606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.231721 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.232294 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:05.232295 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:05.232344 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:05.232369 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:05.232587 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:05.232592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:05.232594 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.232650 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.232734 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260925 10:17:05.233564 22694 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44725
I20260925 10:17:05.233670 22694 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:05.233856 22694 heartbeater.cc:511] Master 127.3.152.254:44725 requested a full tablet report, sending...
I20260925 10:17:05.234277 22375 ts_manager.cc:194] Registered new tserver with Master: c74733bee21a49ca8251186f3af97ec8 (127.3.152.194:35165)
I20260925 10:17:05.234647 22375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.194:34931
I20260925 10:17:05.237742 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22566
I20260925 10:17:05.237826 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:05.238034 3683 external_mini_cluster.cc:1428] Setting key 42e6d254d20cb8b7b008481d8845aa53
I20260925 10:17:05.239017 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--builtin_ntp_servers=127.3.152.212:45757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:05.320510 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:05.320762 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:05.320801 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:05.321949 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:05.322012 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:05.322054 22697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:05.323391 22697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:44725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:05.323665 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:05.323990 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:05.328948 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:05.328945 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:05.328950 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:05.329237 22697 server_base.cc:1034] running on GCE node
I20260925 10:17:05.329463 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:05.329782 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:05.330976 22697 hybrid_clock.cc:648] HybridClock initialized: now 1790331425330959 us; error 25 us; skew 500 ppm
I20260925 10:17:05.332345 22697 webserver.cc:460] Webserver started at http://127.3.152.195:32909/ using document root <none> and password file <none>
I20260925 10:17:05.332626 22697 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:05.332703 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:05.332860 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:05.333966 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "a259f11a93614982ac311f1366a594c1"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "99ccf9f68b8714ae140f21ca4c68fcda"
server_key_iv: "913ba222c18ee12e51778f9bfb00d2d2"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.334338 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "a259f11a93614982ac311f1366a594c1"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "99ccf9f68b8714ae140f21ca4c68fcda"
server_key_iv: "913ba222c18ee12e51778f9bfb00d2d2"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.335671 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:05.336447 22711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.336621 22697 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.336719 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "a259f11a93614982ac311f1366a594c1"
format_stamp: "Formatted at 2026-09-25 10:17:05 on dist-test-slave-dzwq"
server_key: "99ccf9f68b8714ae140f21ca4c68fcda"
server_key_iv: "913ba222c18ee12e51778f9bfb00d2d2"
server_key_version: "encryptionkey@0"
I20260925 10:17:05.336802 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:05.357204 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:05.357594 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:05.357750 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:05.357993 22697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:05.358353 22697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:05.358408 22697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.358448 22697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:05.358476 22697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:05.364365 22697 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:38971
I20260925 10:17:05.364413 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:38971 every 8 connection(s)
I20260925 10:17:05.364759 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:17:05.369894 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.365488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:34791 (local address 127.3.152.254:44725)
0925 10:17:05.365560 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.365563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.365824 (+ 261us) server_negotiation.cc:407] Connection header received
0925 10:17:05.366295 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.366296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.366305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.366359 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:05.366997 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.367504 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.368535 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.368733 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.368791 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.369664 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:05.369665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:05.369666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:05.369670 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:05.369684 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:05.369687 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:05.369737 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.369808 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.369832 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 10:17:05.370103 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.365478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:44725 (local address 127.3.152.195:34791)
0925 10:17:05.365701 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.365737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.366211 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.366413 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.366417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.366425 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:05.366925 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.366929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.367641 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.367641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.368454 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.368456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.368873 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.368874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.368996 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.369514 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:05.369515 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:05.369554 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:05.369578 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:05.369739 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:05.369742 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:05.369744 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.369792 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.369871 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260925 10:17:05.370637 22825 heartbeater.cc:347] Connected to a master server at 127.3.152.254:44725
I20260925 10:17:05.370769 22825 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:05.370968 22825 heartbeater.cc:511] Master 127.3.152.254:44725 requested a full tablet report, sending...
I20260925 10:17:05.371372 22375 ts_manager.cc:194] Registered new tserver with Master: a259f11a93614982ac311f1366a594c1 (127.3.152.195:38971)
I20260925 10:17:05.371750 22375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.195:34791
I20260925 10:17:05.374075 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22697
I20260925 10:17:05.374183 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:05.374400 3683 external_mini_cluster.cc:1428] Setting key b3e6d3dca1ad3e843e250be06642d6f0
I20260925 10:17:05.375353 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:05.379758 22414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.376806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38506 (local address 127.3.152.254:44725)
0925 10:17:05.376845 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.376847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.377015 (+ 168us) server_negotiation.cc:407] Connection header received
0925 10:17:05.377041 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.377041 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.377051 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.377105 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:05.377262 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.377727 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.378430 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.378647 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.378704 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.379417 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:05.379418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:05.379418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:05.379421 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:05.379441 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:05.379444 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:05.379568 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.379632 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.379642 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:17:05.381312 22375 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:05.382867 22375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:05.389364 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.385082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.3.152.195:38971)
0925 10:17:05.385870 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.385875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.385884 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:05.385925 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.385927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.385943 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.386012 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:05.386136 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.387205 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.388401 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.388894 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.388994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.389084 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.389226 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.389269 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":78,"threads_started":1}
I20260925 10:17:05.389390 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.385168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35165 (local address 127.0.0.1:41658)
0925 10:17:05.385367 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.385376 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.385413 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.385757 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.385760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.385763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:05.385829 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.385830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.386848 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.386849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.388059 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.388062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.389091 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.389092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.389149 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.389153 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.389203 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.389284 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":54,"threads_started":1}
I20260925 10:17:05.389389 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.384970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:38971 (local address 127.0.0.1:54426)
0925 10:17:05.385146 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.385186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.385250 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.386027 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.386028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.386031 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:05.386072 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.386073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.387293 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.387294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.388311 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.388313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.388999 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.389000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.389085 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.389099 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.389209 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.389290 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":32,"thread_start_us":59,"threads_started":1}
I20260925 10:17:05.389715 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.385379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.3.152.194:35165)
0925 10:17:05.385551 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.385556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.385566 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:05.385628 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.385630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.385642 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.385750 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:05.385946 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.386756 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.388147 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.388882 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.388954 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.389055 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.389588 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.389631 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260925 10:17:05.390436 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.384974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34183 (local address 127.0.0.1:38274)
0925 10:17:05.385262 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.385275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.385326 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.385915 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.385915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.385917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:05.385947 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.385948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.388015 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.388016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.389150 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.389151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.390198 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.390199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.390251 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.390253 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.390327 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.390368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":98,"threads_started":1}
I20260925 10:17:05.390458 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.385067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.3.152.193:34183)
0925 10:17:05.385577 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.385581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.385589 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:05.385644 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.385646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.385661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.385742 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:05.386038 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.387198 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.389239 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.390004 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.390078 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.390166 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.390342 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.390381 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":97,"threads_started":1}
I20260925 10:17:05.390551 22628 tablet_service.cc:1467] Processing CreateTablet for tablet e659b3916ff34634accfac099c458565 (DEFAULT_TABLE table=test-table [id=bf9ddf71d0cd457198e93a1d5a37a1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:05.390954 22628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b3916ff34634accfac099c458565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:05.391106 22497 tablet_service.cc:1467] Processing CreateTablet for tablet e659b3916ff34634accfac099c458565 (DEFAULT_TABLE table=test-table [id=bf9ddf71d0cd457198e93a1d5a37a1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:05.391424 22497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b3916ff34634accfac099c458565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:05.394008 22839 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Bootstrap starting.
I20260925 10:17:05.394449 22759 tablet_service.cc:1467] Processing CreateTablet for tablet e659b3916ff34634accfac099c458565 (DEFAULT_TABLE table=test-table [id=bf9ddf71d0cd457198e93a1d5a37a1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:05.394891 22839 tablet_bootstrap.cc:654] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:05.394918 22759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e659b3916ff34634accfac099c458565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:05.395244 22839 log.cc:824] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:05.396098 22839 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: No bootstrap required, opened a new log
I20260925 10:17:05.396178 22839 ts_tablet_manager.cc:1397] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:05.396185 22840 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: Bootstrap starting.
I20260925 10:17:05.396893 22840 tablet_bootstrap.cc:654] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:05.397202 22840 log.cc:824] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:05.397930 22840 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: No bootstrap required, opened a new log
I20260925 10:17:05.398018 22840 ts_tablet_manager.cc:1397] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:17:05.398483 22843 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Bootstrap starting.
I20260925 10:17:05.398963 22839 raft_consensus.cc:348] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.399128 22839 raft_consensus.cc:374] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:05.399171 22839 raft_consensus.cc:729] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74733bee21a49ca8251186f3af97ec8, State: Initialized, Role: FOLLOWER
I20260925 10:17:05.399248 22843 tablet_bootstrap.cc:654] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:05.399305 22839 consensus_queue.cc:260] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.399573 22843 log.cc:824] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:05.399614 22694 heartbeater.cc:503] Master 127.3.152.254:44725 was elected leader, sending a full tablet report...
I20260925 10:17:05.399607 22839 ts_tablet_manager.cc:1428] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:17:05.400856 22843 tablet_bootstrap.cc:492] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: No bootstrap required, opened a new log
I20260925 10:17:05.400781 22840 raft_consensus.cc:348] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.400918 22843 ts_tablet_manager.cc:1397] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:05.400929 22840 raft_consensus.cc:374] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:05.400974 22840 raft_consensus.cc:729] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee5c7a9cf454196951f8f21c1db7a69, State: Initialized, Role: FOLLOWER
I20260925 10:17:05.401111 22840 consensus_queue.cc:260] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.401376 22840 ts_tablet_manager.cc:1428] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 10:17:05.401427 22563 heartbeater.cc:503] Master 127.3.152.254:44725 was elected leader, sending a full tablet report...
I20260925 10:17:05.402855 22843 raft_consensus.cc:348] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.402978 22843 raft_consensus.cc:374] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:05.402999 22843 raft_consensus.cc:729] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a259f11a93614982ac311f1366a594c1, State: Initialized, Role: FOLLOWER
I20260925 10:17:05.403116 22843 consensus_queue.cc:260] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.403314 22843 ts_tablet_manager.cc:1428] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:17:05.403421 22825 heartbeater.cc:503] Master 127.3.152.254:44725 was elected leader, sending a full tablet report...
I20260925 10:17:05.417444 22847 raft_consensus.cc:482] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:05.417558 22847 raft_consensus.cc:504] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.417867 22847 leader_election.cc:288] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ee5c7a9cf454196951f8f21c1db7a69 (127.3.152.193:34183), c74733bee21a49ca8251186f3af97ec8 (127.3.152.194:35165)
I20260925 10:17:05.420846 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.418079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34183 (local address 127.3.152.195:53043)
0925 10:17:05.418139 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.418151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.418195 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.418375 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.418377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.418379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:05.418455 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.418458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.419188 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.419189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.420048 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.420050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.420594 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.420670 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.420674 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.420727 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.420771 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":16}
I20260925 10:17:05.420846 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.418152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53043 (local address 127.3.152.193:34183)
0925 10:17:05.418210 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.418212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.418220 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:05.418253 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.418254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.418267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.418334 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:05.418594 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.419108 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420128 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.420508 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420564 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.420571 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.420736 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.420777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:17:05.421017 22838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.418158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:32937 (local address 127.3.152.194:35165)
0925 10:17:05.418198 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.418200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.418386 (+ 186us) server_negotiation.cc:407] Connection header received
0925 10:17:05.418426 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.418427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.418438 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.418498 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:05.418679 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.419164 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420191 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.420588 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420663 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.420670 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.420892 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.420937 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:17:05.421234 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.418082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:35165 (local address 127.3.152.195:32937)
0925 10:17:05.418300 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.418311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:05.418370 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:05.418508 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:05.418509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:05.418510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:05.418597 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.418598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.419251 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.419252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.420099 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:05.420102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.420707 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.420708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:05.420797 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.420800 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:05.421107 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 10:17:05.421145 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":26,"thread_start_us":133,"threads_started":1}
I20260925 10:17:05.421172 22517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b3916ff34634accfac099c458565" candidate_uuid: "a259f11a93614982ac311f1366a594c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" is_pre_election: true
I20260925 10:17:05.421355 22517 raft_consensus.cc:2454] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a259f11a93614982ac311f1366a594c1 in term 0.
I20260925 10:17:05.421571 22648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b3916ff34634accfac099c458565" candidate_uuid: "a259f11a93614982ac311f1366a594c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74733bee21a49ca8251186f3af97ec8" is_pre_election: true
I20260925 10:17:05.421617 22714 leader_election.cc:302] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee5c7a9cf454196951f8f21c1db7a69, a259f11a93614982ac311f1366a594c1; no voters:
I20260925 10:17:05.421695 22648 raft_consensus.cc:2454] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a259f11a93614982ac311f1366a594c1 in term 0.
I20260925 10:17:05.421767 22847 raft_consensus.cc:2781] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:05.421825 22847 raft_consensus.cc:482] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:05.421855 22847 raft_consensus.cc:3037] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:05.422582 22847 raft_consensus.cc:504] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.422734 22847 leader_election.cc:288] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [CANDIDATE]: Term 1 election: Requested vote from peers 0ee5c7a9cf454196951f8f21c1db7a69 (127.3.152.193:34183), c74733bee21a49ca8251186f3af97ec8 (127.3.152.194:35165)
I20260925 10:17:05.422941 22648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b3916ff34634accfac099c458565" candidate_uuid: "a259f11a93614982ac311f1366a594c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c74733bee21a49ca8251186f3af97ec8"
I20260925 10:17:05.422933 22517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e659b3916ff34634accfac099c458565" candidate_uuid: "a259f11a93614982ac311f1366a594c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee5c7a9cf454196951f8f21c1db7a69"
I20260925 10:17:05.423017 22517 raft_consensus.cc:3037] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:05.423017 22648 raft_consensus.cc:3037] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:05.423938 22517 raft_consensus.cc:2454] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a259f11a93614982ac311f1366a594c1 in term 1.
I20260925 10:17:05.423938 22648 raft_consensus.cc:2454] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a259f11a93614982ac311f1366a594c1 in term 1.
I20260925 10:17:05.424170 22713 leader_election.cc:302] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a259f11a93614982ac311f1366a594c1, c74733bee21a49ca8251186f3af97ec8; no voters:
I20260925 10:17:05.424266 22847 raft_consensus.cc:2781] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:05.424422 22847 raft_consensus.cc:686] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 1 LEADER]: Becoming Leader. State: Replica: a259f11a93614982ac311f1366a594c1, State: Running, Role: LEADER
I20260925 10:17:05.424502 22847 consensus_queue.cc:237] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } }
I20260925 10:17:05.425458 22375 catalog_manager.cc:5495] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a259f11a93614982ac311f1366a594c1 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "a259f11a93614982ac311f1366a594c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a259f11a93614982ac311f1366a594c1" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 } health_report { overall_health: UNKNOWN } } }
I20260925 10:17:05.434988 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.432605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.3.152.195:38971)
0925 10:17:05.432680 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:05.432682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:05.432691 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:05.432733 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:05.432734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:05.432751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:05.432830 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:05.432981 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.433450 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.434229 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:05.434428 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:05.434503 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:05.434638 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:05.434734 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:05.434846 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:05.434877 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 10:17:05.434884 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 10:17:05.437014 22648 raft_consensus.cc:1261] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 1 FOLLOWER]: Refusing update from remote peer a259f11a93614982ac311f1366a594c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:05.437014 22517 raft_consensus.cc:1261] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69 [term 1 FOLLOWER]: Refusing update from remote peer a259f11a93614982ac311f1366a594c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:05.437438 22849 consensus_queue.cc:1035] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c74733bee21a49ca8251186f3af97ec8" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:05.437584 22847 consensus_queue.cc:1035] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee5c7a9cf454196951f8f21c1db7a69" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:05.440021 22854 mvcc.cc:205] Tried to move back new op lower bound from 7333197518587084800 to 7333197518539829248. Current Snapshot: MvccSnapshot[applied={T|T < 7333197518587084800}]
I20260925 10:17:05.440470 22852 mvcc.cc:205] Tried to move back new op lower bound from 7333197518587084800 to 7333197518539829248. Current Snapshot: MvccSnapshot[applied={T|T < 7333197518587084800}]
I20260925 10:17:05.440610 22853 mvcc.cc:205] Tried to move back new op lower bound from 7333197518587084800 to 7333197518539829248. Current Snapshot: MvccSnapshot[applied={T|T < 7333197518587084800}]
I20260925 10:17:05.442462 22373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:05.442576 22373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:05.443343 22373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a1dc2592fc0429eb3643c770fe9debc: Sending DeleteTablet for 3 replicas of tablet e659b3916ff34634accfac099c458565
I20260925 10:17:05.443730 22497 tablet_service.cc:1514] Processing DeleteTablet for tablet e659b3916ff34634accfac099c458565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:05 UTC) from {username='slave'} at 127.0.0.1:38274
I20260925 10:17:05.443776 22628 tablet_service.cc:1514] Processing DeleteTablet for tablet e659b3916ff34634accfac099c458565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:05 UTC) from {username='slave'} at 127.0.0.1:41658
I20260925 10:17:05.444018 22759 tablet_service.cc:1514] Processing DeleteTablet for tablet e659b3916ff34634accfac099c458565 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:05 UTC) from {username='slave'} at 127.0.0.1:54426
I20260925 10:17:05.444234 22864 tablet_replica.cc:330] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: stopping tablet replica
I20260925 10:17:05.444329 22864 raft_consensus.cc:2229] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:05.444433 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22435
I20260925 10:17:05.444459 22864 raft_consensus.cc:2258] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:05.444554 22862 tablet_replica.cc:330] T e659b3916ff34634accfac099c458565 P 0ee5c7a9cf454196951f8f21c1db7a69: stopping tablet replica
I20260925 10:17:05.444459 22863 tablet_replica.cc:330] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: stopping tablet replica
I20260925 10:17:05.444761 22863 raft_consensus.cc:2229] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:05.445235 22863 raft_consensus.cc:2258] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:05.445735 22863 ts_tablet_manager.cc:1905] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:05.446727 22864 ts_tablet_manager.cc:1905] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:05.447682 22863 ts_tablet_manager.cc:1918] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:05.447816 22863 log.cc:1196] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/e659b3916ff34634accfac099c458565
I20260925 10:17:05.448173 22863 ts_tablet_manager.cc:1939] T e659b3916ff34634accfac099c458565 P c74733bee21a49ca8251186f3af97ec8: Deleting consensus metadata
I20260925 10:17:05.448657 22360 catalog_manager.cc:4841] TS c74733bee21a49ca8251186f3af97ec8 (127.3.152.194:35165): tablet e659b3916ff34634accfac099c458565 (table test-table [id=bf9ddf71d0cd457198e93a1d5a37a1e1]) successfully deleted
I20260925 10:17:05.448707 22864 ts_tablet_manager.cc:1918] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:05.448772 22864 log.cc:1196] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/e659b3916ff34634accfac099c458565
I20260925 10:17:05.449014 22864 ts_tablet_manager.cc:1939] T e659b3916ff34634accfac099c458565 P a259f11a93614982ac311f1366a594c1: Deleting consensus metadata
I20260925 10:17:05.449455 22360 catalog_manager.cc:4841] TS a259f11a93614982ac311f1366a594c1 (127.3.152.195:38971): tablet e659b3916ff34634accfac099c458565 (table test-table [id=bf9ddf71d0cd457198e93a1d5a37a1e1]) successfully deleted
W20260925 10:17:05.451243 22361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:34183 (error 108)
I20260925 10:17:05.451476 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22566
I20260925 10:17:05.452155 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:05.451625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34183 (local address 127.0.0.1:38282)
0925 10:17:05.452069 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:05.452119 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
W20260925 10:17:05.452294 22361 catalog_manager.cc:4584] TS 0ee5c7a9cf454196951f8f21c1db7a69 (127.3.152.193:34183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e659b3916ff34634accfac099c458565: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34183: connect: Connection refused (error 111)
I20260925 10:17:05.458331 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22697
I20260925 10:17:05.462311 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22344
2026-09-25T10:17:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:05 dist-test-slave-dzwq krb5kdc[22868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:05 dist-test-slave-dzwq krb5kdc[22868](info): set up 2 sockets
Sep 25 10:17:05 dist-test-slave-dzwq krb5kdc[22868](info): commencing operation
krb5kdc: starting...
W20260925 10:17:06.500519 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:06 dist-test-slave-dzwq krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:06.623208 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42925
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:06.704299 22881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:06.704516 22881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:06.704540 22881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:06.705649 22881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:06.705686 22881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:06.705700 22881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:06.705713 22881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:06.705725 22881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:06.707123 22881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:42925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:06.707316 22881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:06.707491 22881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:06.712419 22889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:06.712483 22881 server_base.cc:1034] running on GCE node
W20260925 10:17:06.712428 22886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:06.712428 22887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:06.712891 22881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:06.713187 22881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:06.714367 22881 hybrid_clock.cc:648] HybridClock initialized: now 1790331426714353 us; error 44 us; skew 500 ppm
Sep 25 10:17:06 dist-test-slave-dzwq krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:06.717074 22881 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:06.717543 22881 webserver.cc:460] Webserver started at http://127.3.152.254:44431/ using document root <none> and password file <none>
I20260925 10:17:06.717792 22881 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:06.717845 22881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:06.717936 22881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:06.718943 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "f9e497887f014ff0b983c187d7404983"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "483d60aab4b8ae7bc5aa6204a40e96b7"
server_key_iv: "f8bc922319e2a3a0b11379d5e84aff51"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.719254 22881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "f9e497887f014ff0b983c187d7404983"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "483d60aab4b8ae7bc5aa6204a40e96b7"
server_key_iv: "f8bc922319e2a3a0b11379d5e84aff51"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.720705 22881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 10:17:06.721422 22896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:06.721644 22881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 10:17:06.721716 22881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "f9e497887f014ff0b983c187d7404983"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "483d60aab4b8ae7bc5aa6204a40e96b7"
server_key_iv: "f8bc922319e2a3a0b11379d5e84aff51"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.721771 22881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:06.738495 22881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:06.740976 22881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:06.741099 22881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:06.745093 22881 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:42925
I20260925 10:17:06.745155 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:42925 every 8 connection(s)
I20260925 10:17:06.745503 22881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:06.746373 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:06.748302 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22881
I20260925 10:17:06.748430 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:06.748627 3683 external_mini_cluster.cc:1428] Setting key 62174a809e928451ef80482e8e24bc9d
I20260925 10:17:06.748858 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: Bootstrap starting.
I20260925 10:17:06.749629 22949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:06.749948 22949 log.cc:824] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:06.750716 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: No bootstrap required, opened a new log
I20260925 10:17:06.753099 22949 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } }
I20260925 10:17:06.753300 22949 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:06.753337 22949 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e497887f014ff0b983c187d7404983, State: Initialized, Role: FOLLOWER
I20260925 10:17:06.753455 22949 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } }
I20260925 10:17:06.753526 22949 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:06.753566 22949 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:06.753610 22949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:06.754273 22949 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } }
I20260925 10:17:06.754390 22949 leader_election.cc:302] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9e497887f014ff0b983c187d7404983; no voters:
I20260925 10:17:06.754561 22949 leader_election.cc:288] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:06.754580 22954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:06.754699 22954 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [term 1 LEADER]: Becoming Leader. State: Replica: f9e497887f014ff0b983c187d7404983, State: Running, Role: LEADER
I20260925 10:17:06.754784 22954 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } }
I20260925 10:17:06.754832 22949 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:06.755153 22956 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9e497887f014ff0b983c187d7404983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } } }
I20260925 10:17:06.755257 22956 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:06.755402 22955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e497887f014ff0b983c187d7404983. Latest consensus state: current_term: 1 leader_uuid: "f9e497887f014ff0b983c187d7404983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9e497887f014ff0b983c187d7404983" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42925 } } }
I20260925 10:17:06.755468 22955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:06.755591 22960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:06.756124 22960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 10:17:06 dist-test-slave-dzwq krb5kdc[22868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:06.760192 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:06.749209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.3.152.254:42925)
0925 10:17:06.749422 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:06.749431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:06.749465 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:17:06.750102 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:06.750112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:06.750178 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:06.750317 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:06.751023 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.752449 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.753192 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.753811 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.753922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:06.756793 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:06.756797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:06.756798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:06.756809 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:06.758466 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:06.758891 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:06.758894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:06.758894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:06.758974 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:06.759212 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:06.759214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:06.759214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:06.759475 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:06.759615 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:06.759754 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 10:17:06.759865 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260925 10:17:06.765230 22960 catalog_manager.cc:1347] Generated new cluster ID: 739ca8c137044d32b350efd3eeb40653
I20260925 10:17:06.765285 22960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:06.770233 22960 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:06.771008 22960 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:06.775775 22960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: Generated new TSK 0
I20260925 10:17:06.775959 22960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.152.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
Entry for principal kudu/127.3.152.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab.
I20260925 10:17:06.811442 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:06.918172 22975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:06.918371 22975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:06.918395 22975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:06.919601 22975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:06.919643 22975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:06.919670 22975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:06.921172 22975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.193.keytab
--principal=kudu/127.3.152.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:06.921386 22975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:06.921622 22975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:06.926895 22980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:06.926918 22983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:06.926918 22981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:06.927165 22975 server_base.cc:1034] running on GCE node
I20260925 10:17:06.927385 22975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:06.927634 22975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:06.928812 22975 hybrid_clock.cc:648] HybridClock initialized: now 1790331426928794 us; error 26 us; skew 500 ppm
Sep 25 10:17:06 dist-test-slave-dzwq krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:06.931231 22975 init.cc:377] Logged in from keytab as kudu/127.3.152.193@KRBTEST.COM (short username kudu)
I20260925 10:17:06.931747 22975 webserver.cc:460] Webserver started at http://127.3.152.193:38655/ using document root <none> and password file <none>
I20260925 10:17:06.932055 22975 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:06.932128 22975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:06.932286 22975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:06.933339 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "1e3e280780b38aab6fa4e742177bd16e"
server_key_iv: "82f654af63fade0efe8e9277500ad526"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.933748 22975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "1e3e280780b38aab6fa4e742177bd16e"
server_key_iv: "82f654af63fade0efe8e9277500ad526"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.935052 22975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:17:06.935770 22990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:06.935938 22975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:17:06.936089 22975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897"
format_stamp: "Formatted at 2026-09-25 10:17:06 on dist-test-slave-dzwq"
server_key: "1e3e280780b38aab6fa4e742177bd16e"
server_key_iv: "82f654af63fade0efe8e9277500ad526"
server_key_version: "encryptionkey@0"
I20260925 10:17:06.936172 22975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:06.954507 22975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:06.956914 22975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:06.957098 22975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:06.957372 22975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:06.957736 22975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:06.957793 22975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:06.957834 22975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:06.957877 22975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:06.963941 22975 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:35415
I20260925 10:17:06.964018 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:35415 every 8 connection(s)
I20260925 10:17:06.964335 22975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:17:06.967519 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 22975
I20260925 10:17:06.967672 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:06.967895 3683 external_mini_cluster.cc:1428] Setting key 3414022daa99a081458ecd683d51fb44
Sep 25 10:17:06 dist-test-slave-dzwq krb5kdc[22868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331426, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.193@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:06.974831 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:06.965135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:47993 (local address 127.3.152.254:42925)
0925 10:17:06.965214 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:06.965216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:06.965406 (+ 190us) server_negotiation.cc:407] Connection header received
0925 10:17:06.965878 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:06.965879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:06.965889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:06.965948 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:06.966869 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.967438 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.968851 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.969194 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.969288 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:06.971902 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:06.971904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:06.971904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:06.971912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:06.973414 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:06.973843 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:06.973845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:06.973845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:06.973906 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:06.974167 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:06.974168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:06.974168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:06.974289 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:06.974366 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:06.974644 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 10:17:06.974737 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:17:06.975054 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:06.965089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42925 (local address 127.3.152.193:47993)
0925 10:17:06.965270 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:06.965315 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:06.965794 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:06.966003 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:06.966006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:06.966582 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:06.966781 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:06.966786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.967539 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.967539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:06.968768 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:06.968770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:06.969300 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:06.969301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:06.969407 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:06.970019 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:06.970031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:06.971798 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:06.973517 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:06.973525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:06.973528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:06.973763 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:06.973976 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:06.973976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:06.973977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:06.974093 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:06.974355 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:06.974362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:06.974510 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 10:17:06.974621 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 10:17:06.974795 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 10:17:06.975620 23104 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42925
I20260925 10:17:06.975741 23104 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:06.975932 23104 heartbeater.cc:511] Master 127.3.152.254:42925 requested a full tablet report, sending...
I20260925 10:17:06.976527 22913 ts_manager.cc:194] Registered new tserver with Master: d6f54371f1ff49e1a6f42ab2d3cb4897 (127.3.152.193:35415)
WARNING: no policy specified for kudu/127.3.152.194@KRBTEST.COM; defaulting to no policy
I20260925 10:17:06.977141 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.193@KRBTEST.COM'} at 127.3.152.193:47993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
Entry for principal kudu/127.3.152.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab.
I20260925 10:17:06.994879 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:07.076293 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:07.076532 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:07.076570 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:07.077760 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:07.077834 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:07.077879 23109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:07.079360 23109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.194.keytab
--principal=kudu/127.3.152.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:07.079636 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:07.079887 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:07.085315 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.085482 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.085402 23109 server_base.cc:1034] running on GCE node
W20260925 10:17:07.085285 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.085815 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:07.086046 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:07.087210 23109 hybrid_clock.cc:648] HybridClock initialized: now 1790331427087192 us; error 31 us; skew 500 ppm
Sep 25 10:17:07 dist-test-slave-dzwq krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:07.089938 23109 init.cc:377] Logged in from keytab as kudu/127.3.152.194@KRBTEST.COM (short username kudu)
I20260925 10:17:07.090395 23109 webserver.cc:460] Webserver started at http://127.3.152.194:44057/ using document root <none> and password file <none>
I20260925 10:17:07.090646 23109 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:07.090701 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:07.090873 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:07.092072 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "0704dd561a9c4f08a5503b14e5eb13f4"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "ad796ffd5a5520b6867f7101bc73b949"
server_key_iv: "e9e239455e7d687fb304bafbcd56dd05"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.092427 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "0704dd561a9c4f08a5503b14e5eb13f4"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "ad796ffd5a5520b6867f7101bc73b949"
server_key_iv: "e9e239455e7d687fb304bafbcd56dd05"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.093722 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 10:17:07.094441 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.094627 23109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 10:17:07.094722 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "0704dd561a9c4f08a5503b14e5eb13f4"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "ad796ffd5a5520b6867f7101bc73b949"
server_key_iv: "e9e239455e7d687fb304bafbcd56dd05"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.094796 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:07.118597 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:07.120857 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:07.121023 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:07.121268 23109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:07.121596 23109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:07.121631 23109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.121655 23109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:07.121702 23109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.127880 23109 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:38053
I20260925 10:17:07.127945 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:38053 every 8 connection(s)
I20260925 10:17:07.128327 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:07.130152 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23109
I20260925 10:17:07.130254 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:07.130450 3683 external_mini_cluster.cc:1428] Setting key 875345d7707f0a9cac555b2b96599363
Sep 25 10:17:07 dist-test-slave-dzwq krb5kdc[22868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.194@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:07.138759 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.129123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:60827 (local address 127.3.152.254:42925)
0925 10:17:07.129214 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.129217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.129411 (+ 194us) server_negotiation.cc:407] Connection header received
0925 10:17:07.129885 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.129886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.129896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.129995 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.131218 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.132081 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.133062 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.133383 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.133465 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.135635 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.135637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.135637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:07.135646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.137368 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.137758 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.137760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.137760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.137826 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.138065 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.138066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.138067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.138209 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.138290 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.138541 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.138642 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 10:17:07.138782 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.129081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42925 (local address 127.3.152.194:60827)
0925 10:17:07.129285 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.129327 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.129805 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.130050 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.130053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.130845 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:07.131122 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.131129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.132164 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.132164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.132975 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.132978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.133477 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.133478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.133580 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.134147 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:07.134156 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:07.135539 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:07.137449 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:07.137456 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:07.137457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:07.137677 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:07.137897 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:07.137898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:07.137898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:07.137995 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:07.138283 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:07.138287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:07.138403 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.138487 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.138608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260925 10:17:07.139385 23238 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42925
WARNING: no policy specified for kudu/127.3.152.195@KRBTEST.COM; defaulting to no policy
I20260925 10:17:07.139477 23238 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:07.139668 23238 heartbeater.cc:511] Master 127.3.152.254:42925 requested a full tablet report, sending...
I20260925 10:17:07.140120 22913 ts_manager.cc:194] Registered new tserver with Master: 0704dd561a9c4f08a5503b14e5eb13f4 (127.3.152.194:38053)
I20260925 10:17:07.140575 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.194@KRBTEST.COM'} at 127.3.152.194:60827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
Entry for principal kudu/127.3.152.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab.
I20260925 10:17:07.158586 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:07.241184 23243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:07.241398 23243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:07.241423 23243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:07.242537 23243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:07.242573 23243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:07.242600 23243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:07.244010 23243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.195.keytab
--principal=kudu/127.3.152.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:07.244213 23243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:07.244398 23243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:07.249560 23251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.249467 23249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.249644 23248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.249893 23243 server_base.cc:1034] running on GCE node
I20260925 10:17:07.250110 23243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:07.250357 23243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:07.251533 23243 hybrid_clock.cc:648] HybridClock initialized: now 1790331427251510 us; error 52 us; skew 500 ppm
Sep 25 10:17:07 dist-test-slave-dzwq krb5kdc[22868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:07.254213 23243 init.cc:377] Logged in from keytab as kudu/127.3.152.195@KRBTEST.COM (short username kudu)
I20260925 10:17:07.254782 23243 webserver.cc:460] Webserver started at http://127.3.152.195:40675/ using document root <none> and password file <none>
I20260925 10:17:07.255102 23243 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:07.255173 23243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:07.255334 23243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:07.256610 23243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "2cdeb94588df4bf49ecde505b3dfa4a2"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "fa4aea988fc4f39253b6b4877eb4438f"
server_key_iv: "c595671b01b74c5e0db4b54591e385c9"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.256919 23243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "2cdeb94588df4bf49ecde505b3dfa4a2"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "fa4aea988fc4f39253b6b4877eb4438f"
server_key_iv: "c595671b01b74c5e0db4b54591e385c9"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.258188 23243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:07.258894 23258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.259073 23243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:07.259172 23243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "2cdeb94588df4bf49ecde505b3dfa4a2"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "fa4aea988fc4f39253b6b4877eb4438f"
server_key_iv: "c595671b01b74c5e0db4b54591e385c9"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.259258 23243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:07.290540 23243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:07.292511 23243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:07.292689 23243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:07.292937 23243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:07.293251 23243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:07.293305 23243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.293351 23243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:07.293377 23243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.299634 23243 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:42305
I20260925 10:17:07.299697 23371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:42305 every 8 connection(s)
I20260925 10:17:07.300118 23243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:17:07.303812 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23243
I20260925 10:17:07.303920 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:07.304126 3683 external_mini_cluster.cc:1428] Setting key d060c0b2a5eed9b8799c9ead549e69a5
Sep 25 10:17:07 dist-test-slave-dzwq krb5kdc[22868](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.195@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:07.309670 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.300891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:39841 (local address 127.3.152.254:42925)
0925 10:17:07.300960 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.300963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.301198 (+ 235us) server_negotiation.cc:407] Connection header received
0925 10:17:07.301672 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.301673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.301683 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.301740 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.302723 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.303234 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.304656 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.304922 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.305007 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.307157 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.307158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.307159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:07.307165 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.308437 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.308801 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.308801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.308802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.308852 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.309063 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.309063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.309064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.309166 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.309218 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.309492 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.309579 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:17:07.309876 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.300870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42925 (local address 127.3.152.195:39841)
0925 10:17:07.301068 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.301111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.301581 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.301803 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.301806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.302440 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:07.302639 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.302643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.303343 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.303345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.304565 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.304568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.305015 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.305016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.305178 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.305648 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:07.305656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:07.307074 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:07.308507 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:07.308513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:07.308515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:07.308734 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:07.308910 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:07.308911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:07.308911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:07.309009 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:07.309226 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:07.309230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:07.309341 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.309451 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.309609 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260925 10:17:07.310364 23372 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42925
I20260925 10:17:07.310468 23372 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:07.310678 23372 heartbeater.cc:511] Master 127.3.152.254:42925 requested a full tablet report, sending...
I20260925 10:17:07.311079 22913 ts_manager.cc:194] Registered new tserver with Master: 2cdeb94588df4bf49ecde505b3dfa4a2 (127.3.152.195:42305)
I20260925 10:17:07.311465 22913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.152.195@KRBTEST.COM'} at 127.3.152.195:39841
I20260925 10:17:07.315893 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:07.324352 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.317364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.3.152.254:42925)
0925 10:17:07.317453 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.317456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.317558 (+ 102us) server_negotiation.cc:407] Connection header received
0925 10:17:07.317635 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.317636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.317646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.317708 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.318305 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.318801 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.319549 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.319768 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.319856 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.321201 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.321202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.321203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:07.321210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.322673 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.323174 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.323176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.323176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.323253 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:07.323539 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:07.323540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:07.323540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:07.323854 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.324008 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.324144 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.324229 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:17:07.326157 22913 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:07.327849 22913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:07.334034 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35415 (local address 127.0.0.1:59110)
0925 10:17:07.330324 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.330365 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.330433 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.330693 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.330696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.330698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:07.330777 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.330780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332134 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.332135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.333069 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.333072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.333739 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.333740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.333817 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.333830 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.333905 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.333956 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":58,"thread_start_us":65,"threads_started":1}
I20260925 10:17:07.334045 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.3.152.193:35415)
0925 10:17:07.330527 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.330532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.330542 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:07.330587 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.330588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.330600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.330689 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:07.330870 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332023 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.333174 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.333651 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.333740 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.333802 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.333920 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.333959 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260925 10:17:07.334585 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38053 (local address 127.0.0.1:50420)
0925 10:17:07.330612 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.330623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.330711 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.330960 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.330961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.330963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:07.331011 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.331013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332297 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.332298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.333399 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.333402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.334351 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.334352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.334414 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.334418 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.334469 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.334514 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260925 10:17:07.334844 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:42305 (local address 127.0.0.1:36818)
0925 10:17:07.330793 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.330803 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.330851 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.331158 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.331159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.331161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:07.331201 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.331203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332088 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.332089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.333677 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.333679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.334617 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.334618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.334680 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.334685 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.334735 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.334775 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":62,"thread_start_us":100,"threads_started":1}
I20260925 10:17:07.334913 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.3.152.194:38053)
0925 10:17:07.330545 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.330550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.330701 (+ 151us) server_negotiation.cc:407] Connection header received
0925 10:17:07.330782 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.330784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.330799 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.330875 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:07.331105 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332174 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.333487 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.334193 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.334261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.334316 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.334778 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.334821 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260925 10:17:07.335071 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.330404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.3.152.195:42305)
0925 10:17:07.330574 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.330578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.330932 (+ 354us) server_negotiation.cc:407] Connection header received
0925 10:17:07.330981 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.330984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.331001 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.331072 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:07.331278 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.332002 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.333762 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.334480 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.334535 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.334584 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.334945 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.334998 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260925 10:17:07.335187 23038 tablet_service.cc:1467] Processing CreateTablet for tablet 27a77680771e437081b1ab32ae7c8026 (DEFAULT_TABLE table=test-table [id=9eec58342c554a208201c147d79dd7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:07.335570 23038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a77680771e437081b1ab32ae7c8026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:07.335707 23306 tablet_service.cc:1467] Processing CreateTablet for tablet 27a77680771e437081b1ab32ae7c8026 (DEFAULT_TABLE table=test-table [id=9eec58342c554a208201c147d79dd7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:07.336027 23306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a77680771e437081b1ab32ae7c8026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:07.336503 23172 tablet_service.cc:1467] Processing CreateTablet for tablet 27a77680771e437081b1ab32ae7c8026 (DEFAULT_TABLE table=test-table [id=9eec58342c554a208201c147d79dd7d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:07.336732 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a77680771e437081b1ab32ae7c8026. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:07.339165 23390 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Bootstrap starting.
I20260925 10:17:07.339644 23391 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Bootstrap starting.
I20260925 10:17:07.339915 23390 tablet_bootstrap.cc:654] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:07.340282 23391 tablet_bootstrap.cc:654] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:07.340469 23390 log.cc:824] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:07.340819 23391 log.cc:824] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:07.340826 23389 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Bootstrap starting.
I20260925 10:17:07.341470 23389 tablet_bootstrap.cc:654] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:07.341630 23391 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: No bootstrap required, opened a new log
I20260925 10:17:07.341717 23391 ts_tablet_manager.cc:1397] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:17:07.341871 23389 log.cc:824] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:07.342391 23390 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: No bootstrap required, opened a new log
I20260925 10:17:07.342478 23390 ts_tablet_manager.cc:1397] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 10:17:07.342612 23389 tablet_bootstrap.cc:492] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: No bootstrap required, opened a new log
I20260925 10:17:07.342685 23389 ts_tablet_manager.cc:1397] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 10:17:07.343716 23391 raft_consensus.cc:348] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.343895 23391 raft_consensus.cc:374] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:07.343935 23391 raft_consensus.cc:729] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0704dd561a9c4f08a5503b14e5eb13f4, State: Initialized, Role: FOLLOWER
I20260925 10:17:07.344115 23391 consensus_queue.cc:260] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.344179 23390 raft_consensus.cc:348] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.344365 23390 raft_consensus.cc:374] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:07.344408 23391 ts_tablet_manager.cc:1428] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:17:07.344412 23390 raft_consensus.cc:729] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdeb94588df4bf49ecde505b3dfa4a2, State: Initialized, Role: FOLLOWER
I20260925 10:17:07.344534 23238 heartbeater.cc:503] Master 127.3.152.254:42925 was elected leader, sending a full tablet report...
I20260925 10:17:07.344532 23390 consensus_queue.cc:260] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.344820 23372 heartbeater.cc:503] Master 127.3.152.254:42925 was elected leader, sending a full tablet report...
I20260925 10:17:07.344823 23389 raft_consensus.cc:348] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.345003 23389 raft_consensus.cc:374] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:07.345044 23389 raft_consensus.cc:729] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f54371f1ff49e1a6f42ab2d3cb4897, State: Initialized, Role: FOLLOWER
I20260925 10:17:07.344812 23390 ts_tablet_manager.cc:1428] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 10:17:07.345176 23389 consensus_queue.cc:260] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.345455 23389 ts_tablet_manager.cc:1428] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 10:17:07.345600 23104 heartbeater.cc:503] Master 127.3.152.254:42925 was elected leader, sending a full tablet report...
W20260925 10:17:07.378954 23239 tablet.cc:2369] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 10:17:07.465119 23105 tablet.cc:2369] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:17:07.490410 23396 raft_consensus.cc:482] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:07.490622 23396 raft_consensus.cc:504] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.491129 23396 leader_election.cc:288] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0704dd561a9c4f08a5503b14e5eb13f4 (127.3.152.194:38053), d6f54371f1ff49e1a6f42ab2d3cb4897 (127.3.152.193:35415)
I20260925 10:17:07.494300 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.491404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:42167 (local address 127.3.152.193:35415)
0925 10:17:07.491445 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.491447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.491736 (+ 289us) server_negotiation.cc:407] Connection header received
0925 10:17:07.491803 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.491804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.491813 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.491887 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:07.492098 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.492623 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.493564 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.493936 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.494025 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.494034 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.494184 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.494223 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 10:17:07.494357 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.491408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:40611 (local address 127.3.152.194:38053)
0925 10:17:07.491454 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.491456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.491468 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:17:07.491557 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.491558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.491569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.491660 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:07.491837 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.492466 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.493510 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.493938 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.494039 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.494049 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.494240 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.494280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:17:07.494365 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.491343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:38053 (local address 127.3.152.195:40611)
0925 10:17:07.491387 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.491399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.491451 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.491668 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.491669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.491672 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:07.491756 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.491758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.492566 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.492567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.493399 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.493401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.494037 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.494039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.494152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.494155 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.494231 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.494275 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":23}
I20260925 10:17:07.494549 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.491345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35415 (local address 127.3.152.195:42167)
0925 10:17:07.491656 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.491672 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.491747 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.491889 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.491890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.491891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:07.492013 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.492015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.492720 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.492720 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.493470 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.493472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.494026 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.494027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.494111 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.494114 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.494179 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.494227 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":145,"thread_start_us":121,"threads_started":1}
I20260925 10:17:07.494763 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a77680771e437081b1ab32ae7c8026" candidate_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" is_pre_election: true
I20260925 10:17:07.494978 23192 raft_consensus.cc:2454] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdeb94588df4bf49ecde505b3dfa4a2 in term 0.
I20260925 10:17:07.494977 23058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a77680771e437081b1ab32ae7c8026" candidate_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" is_pre_election: true
I20260925 10:17:07.495136 23058 raft_consensus.cc:2454] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cdeb94588df4bf49ecde505b3dfa4a2 in term 0.
I20260925 10:17:07.495276 23260 leader_election.cc:302] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0704dd561a9c4f08a5503b14e5eb13f4, 2cdeb94588df4bf49ecde505b3dfa4a2; no voters:
I20260925 10:17:07.495424 23396 raft_consensus.cc:2781] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:07.495488 23396 raft_consensus.cc:482] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:07.495507 23396 raft_consensus.cc:3037] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:07.496336 23396 raft_consensus.cc:504] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.496500 23396 leader_election.cc:288] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [CANDIDATE]: Term 1 election: Requested vote from peers 0704dd561a9c4f08a5503b14e5eb13f4 (127.3.152.194:38053), d6f54371f1ff49e1a6f42ab2d3cb4897 (127.3.152.193:35415)
I20260925 10:17:07.496665 23058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a77680771e437081b1ab32ae7c8026" candidate_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897"
I20260925 10:17:07.496742 23058 raft_consensus.cc:3037] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:07.497481 23058 raft_consensus.cc:2454] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdeb94588df4bf49ecde505b3dfa4a2 in term 1.
I20260925 10:17:07.497606 23192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a77680771e437081b1ab32ae7c8026" candidate_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0704dd561a9c4f08a5503b14e5eb13f4"
I20260925 10:17:07.497694 23192 raft_consensus.cc:3037] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:07.497705 23259 leader_election.cc:302] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cdeb94588df4bf49ecde505b3dfa4a2, d6f54371f1ff49e1a6f42ab2d3cb4897; no voters:
I20260925 10:17:07.497856 23396 raft_consensus.cc:2781] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:07.498026 23396 raft_consensus.cc:686] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 1 LEADER]: Becoming Leader. State: Replica: 2cdeb94588df4bf49ecde505b3dfa4a2, State: Running, Role: LEADER
I20260925 10:17:07.498121 23396 consensus_queue.cc:237] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } }
I20260925 10:17:07.498734 23192 raft_consensus.cc:2454] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cdeb94588df4bf49ecde505b3dfa4a2 in term 1.
I20260925 10:17:07.499307 22913 catalog_manager.cc:5495] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cdeb94588df4bf49ecde505b3dfa4a2 (127.3.152.195). New cstate: current_term: 1 leader_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cdeb94588df4bf49ecde505b3dfa4a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 42305 } health_report { overall_health: HEALTHY } } }
I20260925 10:17:07.517280 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.514603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.3.152.195:42305)
0925 10:17:07.514669 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.514671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.514680 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:07.514718 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.514719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.514737 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.514800 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:07.514979 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.515474 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.516389 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.516656 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.516712 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.516853 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:07.516960 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:07.517023 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.517123 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.517140 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 10:17:07.519349 23192 raft_consensus.cc:1261] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 1 FOLLOWER]: Refusing update from remote peer 2cdeb94588df4bf49ecde505b3dfa4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:07.519349 23058 raft_consensus.cc:1261] T 27a77680771e437081b1ab32ae7c8026 P d6f54371f1ff49e1a6f42ab2d3cb4897 [term 1 FOLLOWER]: Refusing update from remote peer 2cdeb94588df4bf49ecde505b3dfa4a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:07.519768 23399 consensus_queue.cc:1035] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0704dd561a9c4f08a5503b14e5eb13f4" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:07.519964 23396 consensus_queue.cc:1035] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f54371f1ff49e1a6f42ab2d3cb4897" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 35415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:07.522152 23402 mvcc.cc:205] Tried to move back new op lower bound from 7333197527115988992 to 7333197527033040896. Current Snapshot: MvccSnapshot[applied={T|T < 7333197527115988992}]
I20260925 10:17:07.522334 23403 mvcc.cc:205] Tried to move back new op lower bound from 7333197527115988992 to 7333197527033040896. Current Snapshot: MvccSnapshot[applied={T|T < 7333197527115988992}]
I20260925 10:17:07.524883 22913 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:07.524988 22913 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:07.525341 23405 mvcc.cc:205] Tried to move back new op lower bound from 7333197527115988992 to 7333197527033040896. Current Snapshot: MvccSnapshot[applied={T|T < 7333197527115988992 or (T in {7333197527115988992})}]
I20260925 10:17:07.525784 22913 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9e497887f014ff0b983c187d7404983: Sending DeleteTablet for 3 replicas of tablet 27a77680771e437081b1ab32ae7c8026
I20260925 10:17:07.526100 23038 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a77680771e437081b1ab32ae7c8026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:07 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:59110
I20260925 10:17:07.526243 23172 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a77680771e437081b1ab32ae7c8026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:07 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:50420
I20260925 10:17:07.526320 23306 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a77680771e437081b1ab32ae7c8026 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:07 UTC) from {username='kudu', principal='kudu/127.3.152.254@KRBTEST.COM'} at 127.0.0.1:36818
I20260925 10:17:07.526530 23413 tablet_replica.cc:330] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: stopping tablet replica
I20260925 10:17:07.526543 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22975
I20260925 10:17:07.527069 23413 raft_consensus.cc:2229] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:07.527066 23412 tablet_replica.cc:330] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: stopping tablet replica
I20260925 10:17:07.527176 23413 raft_consensus.cc:2258] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:07.527323 23412 raft_consensus.cc:2229] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:07.527472 23412 raft_consensus.cc:2258] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:07.527770 23413 ts_tablet_manager.cc:1905] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:07.528009 23412 ts_tablet_manager.cc:1905] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:07.529500 23413 ts_tablet_manager.cc:1918] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:07.529574 23413 log.cc:1196] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/27a77680771e437081b1ab32ae7c8026
I20260925 10:17:07.529853 23413 ts_tablet_manager.cc:1939] T 27a77680771e437081b1ab32ae7c8026 P 2cdeb94588df4bf49ecde505b3dfa4a2: Deleting consensus metadata
I20260925 10:17:07.530191 23412 ts_tablet_manager.cc:1918] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:07.530316 23412 log.cc:1196] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/27a77680771e437081b1ab32ae7c8026
I20260925 10:17:07.530323 22899 catalog_manager.cc:4841] TS 2cdeb94588df4bf49ecde505b3dfa4a2 (127.3.152.195:42305): tablet 27a77680771e437081b1ab32ae7c8026 (table test-table [id=9eec58342c554a208201c147d79dd7d6]) successfully deleted
I20260925 10:17:07.530668 23412 ts_tablet_manager.cc:1939] T 27a77680771e437081b1ab32ae7c8026 P 0704dd561a9c4f08a5503b14e5eb13f4: Deleting consensus metadata
I20260925 10:17:07.531107 22898 catalog_manager.cc:4841] TS 0704dd561a9c4f08a5503b14e5eb13f4 (127.3.152.194:38053): tablet 27a77680771e437081b1ab32ae7c8026 (table test-table [id=9eec58342c554a208201c147d79dd7d6]) successfully deleted
W20260925 10:17:07.532447 22897 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:35415 (error 108)
I20260925 10:17:07.532666 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23109
I20260925 10:17:07.533223 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.532814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:35415 (local address 127.0.0.1:59116)
0925 10:17:07.533145 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.533184 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
W20260925 10:17:07.533406 22897 catalog_manager.cc:4584] TS d6f54371f1ff49e1a6f42ab2d3cb4897 (127.3.152.193:35415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27a77680771e437081b1ab32ae7c8026: Network error: Client connection negotiation failed: client connection to 127.3.152.193:35415: connect: Connection refused (error 111)
I20260925 10:17:07.537456 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23243
I20260925 10:17:07.542837 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 22881
2026-09-25T10:17:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2082 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T10:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:07Z Disabled control of system clock
I20260925 10:17:07.571534 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42509
--webserver_interface=127.3.152.254
--webserver_port=0
--builtin_ntp_servers=127.3.152.212:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:07.653083 23418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:07.653306 23418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:07.653327 23418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:07.654444 23418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:07.654485 23418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:07.654497 23418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:07.654510 23418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:07.654528 23418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:07.655793 23418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:42509
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:07.656059 23418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:07.656322 23418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:07.661090 23423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.661098 23424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.661134 23426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.661098 23418 server_base.cc:1034] running on GCE node
I20260925 10:17:07.661600 23418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:07.661859 23418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:07.663000 23418 hybrid_clock.cc:648] HybridClock initialized: now 1790331427662979 us; error 36 us; skew 500 ppm
I20260925 10:17:07.664229 23418 webserver.cc:460] Webserver started at http://127.3.152.254:45719/ using document root <none> and password file <none>
I20260925 10:17:07.664434 23418 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:07.664476 23418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:07.664620 23418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:07.665643 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "6097f7d73b964ec3a9a27221fb431b46"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "49e9d390c8a32dbdec580c315d0602bc"
server_key_iv: "b43904167359d5833f9ace84efd9a183"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.665936 23418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "6097f7d73b964ec3a9a27221fb431b46"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "49e9d390c8a32dbdec580c315d0602bc"
server_key_iv: "b43904167359d5833f9ace84efd9a183"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.667124 23418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:07.667809 23432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.668025 23418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:07.668121 23418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "6097f7d73b964ec3a9a27221fb431b46"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "49e9d390c8a32dbdec580c315d0602bc"
server_key_iv: "b43904167359d5833f9ace84efd9a183"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.668202 23418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:07.680757 23418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:07.681104 23418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:07.681258 23418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:07.685364 23418 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:42509
I20260925 10:17:07.685413 23484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:42509 every 8 connection(s)
I20260925 10:17:07.685774 23418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:07.686733 23485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:07.688941 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: Bootstrap starting.
I20260925 10:17:07.689476 23485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:07.689714 23485 log.cc:824] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:07.690347 23485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: No bootstrap required, opened a new log
I20260925 10:17:07.691871 23485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } }
I20260925 10:17:07.692076 23485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:07.692113 23485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6097f7d73b964ec3a9a27221fb431b46, State: Initialized, Role: FOLLOWER
I20260925 10:17:07.692229 23485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } }
I20260925 10:17:07.692281 23485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:07.692301 23485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:07.692333 23485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:07.692790 23485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } }
I20260925 10:17:07.692870 23485 leader_election.cc:302] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6097f7d73b964ec3a9a27221fb431b46; no voters:
I20260925 10:17:07.692991 23485 leader_election.cc:288] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:07.693068 23488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:07.693244 23488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [term 1 LEADER]: Becoming Leader. State: Replica: 6097f7d73b964ec3a9a27221fb431b46, State: Running, Role: LEADER
I20260925 10:17:07.693274 23485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:07.693344 23488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } }
I20260925 10:17:07.693753 23490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6097f7d73b964ec3a9a27221fb431b46. Latest consensus state: current_term: 1 leader_uuid: "6097f7d73b964ec3a9a27221fb431b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } } }
I20260925 10:17:07.693943 23490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:07.693832 23489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6097f7d73b964ec3a9a27221fb431b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6097f7d73b964ec3a9a27221fb431b46" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 42509 } } }
I20260925 10:17:07.694120 23489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:07.694232 23497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:07.694749 23497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:07.695919 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23418
I20260925 10:17:07.696046 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:07.696269 3683 external_mini_cluster.cc:1428] Setting key 63c3f9bae2890797c672261b772c2896
I20260925 10:17:07.696333 23497 catalog_manager.cc:1347] Generated new cluster ID: 067342db85214491b2aeae338d0b1260
I20260925 10:17:07.696377 23497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:07.705854 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.696777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.3.152.254:42509)
0925 10:17:07.700359 (+ 3582us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.700369 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.700403 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:17:07.700540 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.700548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.700628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.700784 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.701027 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.702528 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.703584 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.704226 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.704348 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.705092 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.705096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.705096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:07.705323 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.705366 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:07.705382 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.705452 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.705519 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.705599 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":72,"threads_started":1}
I20260925 10:17:07.712363 23497 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:07.712884 23497 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:07.714974 23497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: Generated new TSK 0
I20260925 10:17:07.715152 23497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:07.716635 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.193:0
--local_ip_for_outbound_sockets=127.3.152.193
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--builtin_ntp_servers=127.3.152.212:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:07.797853 23509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:07.798065 23509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:07.798087 23509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:07.799355 23509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:07.799396 23509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:07.799422 23509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.193
I20260925 10:17:07.800920 23509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.152.193
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.193
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:07.801167 23509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:07.801486 23509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:07.806511 23514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.806541 23515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.806567 23517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.807024 23509 server_base.cc:1034] running on GCE node
I20260925 10:17:07.807224 23509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:07.807458 23509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:07.808614 23509 hybrid_clock.cc:648] HybridClock initialized: now 1790331427808592 us; error 37 us; skew 500 ppm
I20260925 10:17:07.810048 23509 webserver.cc:460] Webserver started at http://127.3.152.193:46385/ using document root <none> and password file <none>
I20260925 10:17:07.810313 23509 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:07.810379 23509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:07.810509 23509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:07.811935 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data/instance:
uuid: "242675df5ab44e30a6cb27a5da78878e"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "09196b7a1c4655fefcdabd4efc862842"
server_key_iv: "0e7070e328df79afd42db9581fbdf403"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.812393 23509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance:
uuid: "242675df5ab44e30a6cb27a5da78878e"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "09196b7a1c4655fefcdabd4efc862842"
server_key_iv: "0e7070e328df79afd42db9581fbdf403"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.814043 23509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:07.814945 23523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.815143 23509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:07.815230 23509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal
uuid: "242675df5ab44e30a6cb27a5da78878e"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "09196b7a1c4655fefcdabd4efc862842"
server_key_iv: "0e7070e328df79afd42db9581fbdf403"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.815318 23509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:07.828194 23509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:07.828512 23509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:07.828667 23509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:07.828905 23509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:07.829257 23509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:07.829311 23509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.829346 23509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:07.829375 23509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.834964 23509 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.193:34987
I20260925 10:17:07.835007 23636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.193:34987 every 8 connection(s)
I20260925 10:17:07.835371 23509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/data/info.pb
I20260925 10:17:07.840574 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.836240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.193:39049 (local address 127.3.152.254:42509)
0925 10:17:07.836313 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.836316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.836550 (+ 234us) server_negotiation.cc:407] Connection header received
0925 10:17:07.837020 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.837021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.837035 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.837087 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.837748 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.838253 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.839277 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.839468 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.839520 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.840339 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.840341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.840341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:07.840345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.840367 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:07.840371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.840418 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.840487 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.840515 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 10:17:07.840790 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.836230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42509 (local address 127.3.152.193:39049)
0925 10:17:07.836423 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.836460 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.836937 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.837144 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.837147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.837155 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:07.837671 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.837675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.838358 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.838359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.839193 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.839195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.839543 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.839544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.839654 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.840207 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:07.840207 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:07.840235 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:07.840259 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:07.840421 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:07.840423 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:07.840425 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.840472 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.840550 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260925 10:17:07.841341 23637 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42509
I20260925 10:17:07.841445 23637 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:07.841658 23637 heartbeater.cc:511] Master 127.3.152.254:42509 requested a full tablet report, sending...
I20260925 10:17:07.841912 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23509
I20260925 10:17:07.842024 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-0/wal/instance
I20260925 10:17:07.842233 3683 external_mini_cluster.cc:1428] Setting key 23334150366c7fd4d6f09764d6ac0268
I20260925 10:17:07.842224 23449 ts_manager.cc:194] Registered new tserver with Master: 242675df5ab44e30a6cb27a5da78878e (127.3.152.193:34987)
I20260925 10:17:07.842895 23449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.193:39049
I20260925 10:17:07.843458 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.194:0
--local_ip_for_outbound_sockets=127.3.152.194
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--builtin_ntp_servers=127.3.152.212:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:07.924957 23640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:07.925244 23640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:07.925280 23640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:07.926679 23640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:07.926728 23640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:07.926755 23640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.194
I20260925 10:17:07.928241 23640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.152.194
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.194
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:07.928438 23640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:07.928699 23640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:07.933879 23646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.933840 23648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:07.933823 23645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:07.933892 23640 server_base.cc:1034] running on GCE node
I20260925 10:17:07.934221 23640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:07.934481 23640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:07.935638 23640 hybrid_clock.cc:648] HybridClock initialized: now 1790331427935610 us; error 41 us; skew 500 ppm
I20260925 10:17:07.937088 23640 webserver.cc:460] Webserver started at http://127.3.152.194:46653/ using document root <none> and password file <none>
I20260925 10:17:07.937366 23640 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:07.937413 23640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:07.937597 23640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:07.938704 23640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data/instance:
uuid: "745cf51648994b44a8776edcac6f4dd5"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "58e559b07410ac6351ecfdf49545abe2"
server_key_iv: "906d0e0d02317acd374021d33663379e"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.939078 23640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance:
uuid: "745cf51648994b44a8776edcac6f4dd5"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "58e559b07410ac6351ecfdf49545abe2"
server_key_iv: "906d0e0d02317acd374021d33663379e"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.940554 23640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:17:07.941298 23654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.941475 23640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:07.941583 23640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal
uuid: "745cf51648994b44a8776edcac6f4dd5"
format_stamp: "Formatted at 2026-09-25 10:17:07 on dist-test-slave-dzwq"
server_key: "58e559b07410ac6351ecfdf49545abe2"
server_key_iv: "906d0e0d02317acd374021d33663379e"
server_key_version: "encryptionkey@0"
I20260925 10:17:07.941663 23640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:07.952517 23640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:07.952842 23640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:07.952997 23640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:07.953243 23640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:07.953581 23640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:07.953639 23640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.953681 23640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:07.953706 23640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:07.959553 23640 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.194:42873
I20260925 10:17:07.959626 23767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.194:42873 every 8 connection(s)
I20260925 10:17:07.960043 23640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/data/info.pb
I20260925 10:17:07.965341 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.960825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:40039 (local address 127.3.152.254:42509)
0925 10:17:07.960898 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:07.960901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:07.961221 (+ 320us) server_negotiation.cc:407] Connection header received
0925 10:17:07.961706 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:07.961707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:07.961721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:07.961776 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:07.962410 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.962896 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.963943 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.964229 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.964284 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.965100 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:07.965101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:07.965101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:07.965105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:07.965120 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:07.965124 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:07.965165 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:07.965253 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 10:17:07.965279 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:17:07.965529 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:07.960820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42509 (local address 127.3.152.194:40039)
0925 10:17:07.961083 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:07.961121 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:07.961617 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:07.961829 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:07.961833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:07.961841 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:07.962341 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.962345 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.962990 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.962991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.963867 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:07.963869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:07.964309 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:07.964310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:07.964420 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:07.964958 (+ 538us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:07.964958 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:07.964987 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:07.965009 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:07.965179 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:07.965183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:07.965186 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:07.965239 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:17:07.965320 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":102,"threads_started":1}
I20260925 10:17:07.966140 23768 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42509
I20260925 10:17:07.966245 23768 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:07.966434 23768 heartbeater.cc:511] Master 127.3.152.254:42509 requested a full tablet report, sending...
I20260925 10:17:07.966871 23449 ts_manager.cc:194] Registered new tserver with Master: 745cf51648994b44a8776edcac6f4dd5 (127.3.152.194:42873)
I20260925 10:17:07.967255 23449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.194:40039
I20260925 10:17:07.968427 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23640
I20260925 10:17:07.968508 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal/instance
I20260925 10:17:07.968658 3683 external_mini_cluster.cc:1428] Setting key 72cf739a5e3a86497bc6d7debf6f81c8
I20260925 10:17:07.969511 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.152.195:0
--local_ip_for_outbound_sockets=127.3.152.195
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--builtin_ntp_servers=127.3.152.212:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 10:17:08.050684 23771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:08.050899 23771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:08.050921 23771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:08.052004 23771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:08.052038 23771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:08.052063 23771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.152.195
I20260925 10:17:08.053363 23771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.152.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.152.195
--webserver_port=0
--tserver_master_addrs=127.3.152.254:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.152.195
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:08.053553 23771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:08.053797 23771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:08.058970 23779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:08.058997 23776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:08.058979 23777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:08.059247 23771 server_base.cc:1034] running on GCE node
I20260925 10:17:08.059423 23771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:08.059653 23771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:08.060825 23771 hybrid_clock.cc:648] HybridClock initialized: now 1790331428060813 us; error 39 us; skew 500 ppm
I20260925 10:17:08.062029 23771 webserver.cc:460] Webserver started at http://127.3.152.195:40193/ using document root <none> and password file <none>
I20260925 10:17:08.062268 23771 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:08.062338 23771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:08.062492 23771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:08.063668 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data/instance:
uuid: "93cb092f0c9d4d67967edc06e56a53a2"
format_stamp: "Formatted at 2026-09-25 10:17:08 on dist-test-slave-dzwq"
server_key: "d897744b35064856b0783d366976a97b"
server_key_iv: "1618dff69d569315785344eda21cd248"
server_key_version: "encryptionkey@0"
I20260925 10:17:08.064096 23771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance:
uuid: "93cb092f0c9d4d67967edc06e56a53a2"
format_stamp: "Formatted at 2026-09-25 10:17:08 on dist-test-slave-dzwq"
server_key: "d897744b35064856b0783d366976a97b"
server_key_iv: "1618dff69d569315785344eda21cd248"
server_key_version: "encryptionkey@0"
I20260925 10:17:08.065451 23771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:08.066239 23785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:08.066431 23771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:08.066589 23771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal
uuid: "93cb092f0c9d4d67967edc06e56a53a2"
format_stamp: "Formatted at 2026-09-25 10:17:08 on dist-test-slave-dzwq"
server_key: "d897744b35064856b0783d366976a97b"
server_key_iv: "1618dff69d569315785344eda21cd248"
server_key_version: "encryptionkey@0"
I20260925 10:17:08.066670 23771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:08.097457 23771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:08.097820 23771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:08.097985 23771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:08.098234 23771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 10:17:08.098593 23771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 10:17:08.098649 23771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:08.098690 23771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 10:17:08.098718 23771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:08.104537 23771 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.195:35393
I20260925 10:17:08.104576 23898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.195:35393 every 8 connection(s)
I20260925 10:17:08.104914 23771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/data/info.pb
I20260925 10:17:08.109964 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.105679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.195:53181 (local address 127.3.152.254:42509)
0925 10:17:08.105731 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.105733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.105954 (+ 221us) server_negotiation.cc:407] Connection header received
0925 10:17:08.106433 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.106434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.106443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.106514 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:08.107164 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.107660 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.108654 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.108842 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.108896 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.109722 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:08.109723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:08.109723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:08.109728 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:08.109742 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:08.109746 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:08.109789 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.109874 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.109901 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:17:08.110153 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.105637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:42509 (local address 127.3.152.195:53181)
0925 10:17:08.105819 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.105860 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.106349 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.106570 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.106573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.106581 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:08.107089 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.107094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.107748 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.107749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.108571 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.108574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.108918 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.108919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.109035 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.109579 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 10:17:08.109579 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:08.109608 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 10:17:08.109631 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:08.109801 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:08.109805 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:08.109807 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.109860 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.109942 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260925 10:17:08.110677 23899 heartbeater.cc:347] Connected to a master server at 127.3.152.254:42509
I20260925 10:17:08.110792 23899 heartbeater.cc:464] Registering TS with master...
I20260925 10:17:08.110977 23899 heartbeater.cc:511] Master 127.3.152.254:42509 requested a full tablet report, sending...
I20260925 10:17:08.111359 23449 ts_manager.cc:194] Registered new tserver with Master: 93cb092f0c9d4d67967edc06e56a53a2 (127.3.152.195:35393)
I20260925 10:17:08.111742 23449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.152.195:53181
I20260925 10:17:08.114848 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23771
I20260925 10:17:08.114936 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal/instance
I20260925 10:17:08.115142 3683 external_mini_cluster.cc:1428] Setting key f2bd5e611f2c627c9a52171c435c8351
I20260925 10:17:08.116247 3683 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 10:17:08.120839 23508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.117737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.3.152.254:42509)
0925 10:17:08.117782 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.117784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.117918 (+ 134us) server_negotiation.cc:407] Connection header received
0925 10:17:08.117990 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.117991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.118001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.118050 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:08.118237 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.118785 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.119702 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.119878 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.119927 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.120591 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:08.120592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:08.120593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:08.120597 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:08.120608 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:08.120612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:08.120655 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.120758 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.120781 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:08.122495 23449 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 10:17:08.124069 23449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 10:17:08.131016 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.194:42873 (local address 127.0.0.1:54782)
0925 10:17:08.126798 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.126807 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.126845 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.127266 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.127267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.127268 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:08.127312 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.127314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.128264 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.128266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.129717 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.129719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.130705 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.130706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.130784 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.130801 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.130881 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.130930 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":65,"thread_start_us":71,"threads_started":1}
I20260925 10:17:08.131060 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.3.152.194:42873)
0925 10:17:08.127092 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.127096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.127105 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:08.127148 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.127151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.127166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.127268 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:08.127371 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.128142 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.129818 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.130610 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.130684 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.130758 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.130917 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.130971 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":131,"threads_started":1}
I20260925 10:17:08.131680 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:35393 (local address 127.0.0.1:43562)
0925 10:17:08.126520 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.126565 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.126639 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.128280 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.128281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.128283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:08.128329 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.128330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.129452 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.129454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.130485 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.130488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.131474 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.131475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.131540 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.131544 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.131574 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.131612 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260925 10:17:08.132064 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.3.152.195:35393)
0925 10:17:08.128100 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.128105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.128117 (+ 12us) server_negotiation.cc:407] Connection header received
0925 10:17:08.128161 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.128163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.128181 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.128275 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:08.128390 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.129351 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.130589 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.131373 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.131788 (+ 415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.131884 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.131911 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.131954 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1370,"thread_start_us":80,"threads_started":1}
I20260925 10:17:08.132001 23702 tablet_service.cc:1467] Processing CreateTablet for tablet 4df9b87659dc40c18ad0f187de726962 (DEFAULT_TABLE table=test-table [id=416733135f774051914bb4ad5b29bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:08.132198 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34987 (local address 127.0.0.1:45762)
0925 10:17:08.126661 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.126670 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.126714 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.127077 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.127081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.127085 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:08.127162 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.127164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.128270 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.128271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.129712 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.129714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.131904 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.131905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.132008 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.132015 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.132055 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.132105 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":63,"thread_start_us":114,"threads_started":1}
I20260925 10:17:08.132411 23702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df9b87659dc40c18ad0f187de726962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:08.132668 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.126670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.3.152.193:34987)
0925 10:17:08.126866 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.126872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.126883 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:08.126927 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.126929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.126945 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.127257 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:08.127325 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.128156 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.130792 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.131791 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.132395 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.132504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.132531 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.132574 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260925 10:17:08.133282 23571 tablet_service.cc:1467] Processing CreateTablet for tablet 4df9b87659dc40c18ad0f187de726962 (DEFAULT_TABLE table=test-table [id=416733135f774051914bb4ad5b29bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:08.133607 23571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df9b87659dc40c18ad0f187de726962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:08.135620 23833 tablet_service.cc:1467] Processing CreateTablet for tablet 4df9b87659dc40c18ad0f187de726962 (DEFAULT_TABLE table=test-table [id=416733135f774051914bb4ad5b29bd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 10:17:08.135931 23833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4df9b87659dc40c18ad0f187de726962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:08.136178 23913 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Bootstrap starting.
I20260925 10:17:08.136742 23914 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: Bootstrap starting.
I20260925 10:17:08.136958 23913 tablet_bootstrap.cc:654] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:08.137317 23913 log.cc:824] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:08.137519 23914 tablet_bootstrap.cc:654] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:08.137838 23914 log.cc:824] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:08.138239 23913 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: No bootstrap required, opened a new log
I20260925 10:17:08.138322 23913 ts_tablet_manager.cc:1397] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:08.138877 23914 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: No bootstrap required, opened a new log
I20260925 10:17:08.138952 23914 ts_tablet_manager.cc:1397] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:08.139358 23917 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Bootstrap starting.
I20260925 10:17:08.140064 23917 tablet_bootstrap.cc:654] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:08.140393 23917 log.cc:824] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:08.141229 23917 tablet_bootstrap.cc:492] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: No bootstrap required, opened a new log
I20260925 10:17:08.141306 23917 ts_tablet_manager.cc:1397] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 10:17:08.141227 23913 raft_consensus.cc:348] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.141417 23913 raft_consensus.cc:374] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:08.141464 23913 raft_consensus.cc:729] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 745cf51648994b44a8776edcac6f4dd5, State: Initialized, Role: FOLLOWER
I20260925 10:17:08.141426 23914 raft_consensus.cc:348] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.141592 23914 raft_consensus.cc:374] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:08.141637 23914 raft_consensus.cc:729] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242675df5ab44e30a6cb27a5da78878e, State: Initialized, Role: FOLLOWER
I20260925 10:17:08.141639 23913 consensus_queue.cc:260] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.141767 23914 consensus_queue.cc:260] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.141935 23913 ts_tablet_manager.cc:1428] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 10:17:08.142022 23914 ts_tablet_manager.cc:1428] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 10:17:08.142081 23637 heartbeater.cc:503] Master 127.3.152.254:42509 was elected leader, sending a full tablet report...
I20260925 10:17:08.142092 23768 heartbeater.cc:503] Master 127.3.152.254:42509 was elected leader, sending a full tablet report...
I20260925 10:17:08.143889 23917 raft_consensus.cc:348] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.144096 23917 raft_consensus.cc:374] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:08.144156 23917 raft_consensus.cc:729] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cb092f0c9d4d67967edc06e56a53a2, State: Initialized, Role: FOLLOWER
I20260925 10:17:08.144285 23917 consensus_queue.cc:260] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.144567 23917 ts_tablet_manager.cc:1428] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 10:17:08.144619 23899 heartbeater.cc:503] Master 127.3.152.254:42509 was elected leader, sending a full tablet report...
W20260925 10:17:08.210628 23769 tablet.cc:2369] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 10:17:08.251339 23919 raft_consensus.cc:482] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 10:17:08.251477 23919 raft_consensus.cc:504] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.251866 23919 leader_election.cc:288] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93cb092f0c9d4d67967edc06e56a53a2 (127.3.152.195:35393), 242675df5ab44e30a6cb27a5da78878e (127.3.152.193:34987)
I20260925 10:17:08.254796 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.252266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:40999 (local address 127.3.152.195:35393)
0925 10:17:08.252313 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.252315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.252333 (+ 18us) server_negotiation.cc:407] Connection header received
0925 10:17:08.252361 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.252362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.252370 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.252459 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:08.252616 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.253101 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254067 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.254429 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.254544 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.254691 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.254734 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:17:08.254796 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.252158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.195:35393 (local address 127.3.152.194:40999)
0925 10:17:08.252206 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.252218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.252264 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.252464 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.252465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.252467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:08.252541 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.252544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.253198 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.253199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.253946 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.253948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.254532 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.254607 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.254610 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.254673 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.254718 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 10:17:08.255090 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.252246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.152.194:42443 (local address 127.3.152.193:34987)
0925 10:17:08.252325 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.252328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.252592 (+ 264us) server_negotiation.cc:407] Connection header received
0925 10:17:08.252665 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.252666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.252678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.252718 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 10:17:08.252877 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.253354 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254310 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.254677 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254755 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.254762 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.254966 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.255019 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 10:17:08.255178 23853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df9b87659dc40c18ad0f187de726962" candidate_uuid: "745cf51648994b44a8776edcac6f4dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cb092f0c9d4d67967edc06e56a53a2" is_pre_election: true
I20260925 10:17:08.255309 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.252167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34987 (local address 127.3.152.194:42443)
0925 10:17:08.252506 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.252527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:08.252597 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:08.252729 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:08.252730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:08.252731 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 10:17:08.252802 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.252809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.253440 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.253441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.254213 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:08.254216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.254775 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.254776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:08.254871 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.254875 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 10:17:08.255187 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 10:17:08.255231 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":171,"thread_start_us":116,"threads_started":1}
I20260925 10:17:08.255339 23853 raft_consensus.cc:2454] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745cf51648994b44a8776edcac6f4dd5 in term 0.
I20260925 10:17:08.255569 23655 leader_election.cc:302] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 745cf51648994b44a8776edcac6f4dd5, 93cb092f0c9d4d67967edc06e56a53a2; no voters:
I20260925 10:17:08.255698 23919 raft_consensus.cc:2781] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 10:17:08.255687 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df9b87659dc40c18ad0f187de726962" candidate_uuid: "745cf51648994b44a8776edcac6f4dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242675df5ab44e30a6cb27a5da78878e" is_pre_election: true
I20260925 10:17:08.255762 23919 raft_consensus.cc:482] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 10:17:08.255798 23919 raft_consensus.cc:3037] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:08.255838 23591 raft_consensus.cc:2454] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 745cf51648994b44a8776edcac6f4dd5 in term 0.
I20260925 10:17:08.256608 23919 raft_consensus.cc:504] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.256754 23919 leader_election.cc:288] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [CANDIDATE]: Term 1 election: Requested vote from peers 93cb092f0c9d4d67967edc06e56a53a2 (127.3.152.195:35393), 242675df5ab44e30a6cb27a5da78878e (127.3.152.193:34987)
I20260925 10:17:08.256908 23853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df9b87659dc40c18ad0f187de726962" candidate_uuid: "745cf51648994b44a8776edcac6f4dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93cb092f0c9d4d67967edc06e56a53a2"
I20260925 10:17:08.256927 23591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4df9b87659dc40c18ad0f187de726962" candidate_uuid: "745cf51648994b44a8776edcac6f4dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242675df5ab44e30a6cb27a5da78878e"
I20260925 10:17:08.256992 23853 raft_consensus.cc:3037] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:08.256992 23591 raft_consensus.cc:3037] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:08.257915 23853 raft_consensus.cc:2454] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745cf51648994b44a8776edcac6f4dd5 in term 1.
I20260925 10:17:08.257921 23591 raft_consensus.cc:2454] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 745cf51648994b44a8776edcac6f4dd5 in term 1.
I20260925 10:17:08.258102 23658 leader_election.cc:302] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242675df5ab44e30a6cb27a5da78878e, 745cf51648994b44a8776edcac6f4dd5; no voters:
I20260925 10:17:08.258200 23919 raft_consensus.cc:2781] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:08.258383 23919 raft_consensus.cc:686] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 745cf51648994b44a8776edcac6f4dd5, State: Running, Role: LEADER
I20260925 10:17:08.258574 23919 consensus_queue.cc:237] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } }
I20260925 10:17:08.259421 23448 catalog_manager.cc:5495] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 745cf51648994b44a8776edcac6f4dd5 (127.3.152.194). New cstate: current_term: 1 leader_uuid: "745cf51648994b44a8776edcac6f4dd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "745cf51648994b44a8776edcac6f4dd5" member_type: VOTER last_known_addr { host: "127.3.152.194" port: 42873 } health_report { overall_health: HEALTHY } } }
I20260925 10:17:08.276554 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.273858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.3.152.194:42873)
0925 10:17:08.273968 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:08.273970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:08.274050 (+ 80us) server_negotiation.cc:407] Connection header received
0925 10:17:08.274149 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:08.274150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:08.274172 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:08.274259 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:08.274404 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.274881 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.275687 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:08.275906 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:08.275999 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:08.276178 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:08.276292 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:08.276387 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:08.276410 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 10:17:08.276432 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 10:17:08.278395 23853 raft_consensus.cc:1261] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 1 FOLLOWER]: Refusing update from remote peer 745cf51648994b44a8776edcac6f4dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:08.278419 23591 raft_consensus.cc:1261] T 4df9b87659dc40c18ad0f187de726962 P 242675df5ab44e30a6cb27a5da78878e [term 1 FOLLOWER]: Refusing update from remote peer 745cf51648994b44a8776edcac6f4dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:08.278815 23923 consensus_queue.cc:1035] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93cb092f0c9d4d67967edc06e56a53a2" member_type: VOTER last_known_addr { host: "127.3.152.195" port: 35393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:08.278931 23919 consensus_queue.cc:1035] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "242675df5ab44e30a6cb27a5da78878e" member_type: VOTER last_known_addr { host: "127.3.152.193" port: 34987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:08.281886 23928 mvcc.cc:205] Tried to move back new op lower bound from 7333197530225631232 to 7333197530147741696. Current Snapshot: MvccSnapshot[applied={T|T < 7333197530225631232}]
I20260925 10:17:08.284186 23447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:08.284307 23447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 10:17:08.285136 23447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6097f7d73b964ec3a9a27221fb431b46: Sending DeleteTablet for 3 replicas of tablet 4df9b87659dc40c18ad0f187de726962
I20260925 10:17:08.285482 23571 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df9b87659dc40c18ad0f187de726962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:08 UTC) from {username='slave'} at 127.0.0.1:45762
I20260925 10:17:08.285516 23833 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df9b87659dc40c18ad0f187de726962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:08 UTC) from {username='slave'} at 127.0.0.1:43562
I20260925 10:17:08.285614 23702 tablet_service.cc:1514] Processing DeleteTablet for tablet 4df9b87659dc40c18ad0f187de726962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 10:17:08 UTC) from {username='slave'} at 127.0.0.1:54782
I20260925 10:17:08.285745 23937 tablet_replica.cc:330] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: stopping tablet replica
I20260925 10:17:08.285794 23938 tablet_replica.cc:330] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: stopping tablet replica
I20260925 10:17:08.285849 23937 raft_consensus.cc:2229] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 10:17:08.285888 23938 raft_consensus.cc:2229] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 10:17:08.285944 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23509
I20260925 10:17:08.285956 23937 raft_consensus.cc:2258] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:08.286003 23938 raft_consensus.cc:2258] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 10:17:08.286501 23938 ts_tablet_manager.cc:1905] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:08.287282 23937 ts_tablet_manager.cc:1905] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 10:17:08.288204 23938 ts_tablet_manager.cc:1918] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:08.288285 23938 log.cc:1196] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-1/wal/wals/4df9b87659dc40c18ad0f187de726962
I20260925 10:17:08.288538 23938 ts_tablet_manager.cc:1939] T 4df9b87659dc40c18ad0f187de726962 P 745cf51648994b44a8776edcac6f4dd5: Deleting consensus metadata
I20260925 10:17:08.288916 23435 catalog_manager.cc:4841] TS 745cf51648994b44a8776edcac6f4dd5 (127.3.152.194:42873): tablet 4df9b87659dc40c18ad0f187de726962 (table test-table [id=416733135f774051914bb4ad5b29bd99]) successfully deleted
I20260925 10:17:08.289232 23937 ts_tablet_manager.cc:1918] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 10:17:08.289307 23937 log.cc:1196] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Deleting WAL directory at /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790331159020724-3683-0/minicluster-data/ts-2/wal/wals/4df9b87659dc40c18ad0f187de726962
I20260925 10:17:08.289548 23937 ts_tablet_manager.cc:1939] T 4df9b87659dc40c18ad0f187de726962 P 93cb092f0c9d4d67967edc06e56a53a2: Deleting consensus metadata
I20260925 10:17:08.289932 23433 catalog_manager.cc:4841] TS 93cb092f0c9d4d67967edc06e56a53a2 (127.3.152.195:35393): tablet 4df9b87659dc40c18ad0f187de726962 (table test-table [id=416733135f774051914bb4ad5b29bd99]) successfully deleted
W20260925 10:17:08.293123 23436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.152.193:34987 (error 108)
I20260925 10:17:08.293290 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23640
I20260925 10:17:08.294269 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:08.294116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.193:34987 (local address 127.0.0.1:45768)
0925 10:17:08.294183 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:08.294229 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 10:17:08.294389 23436 catalog_manager.cc:4584] TS 242675df5ab44e30a6cb27a5da78878e (127.3.152.193:34987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4df9b87659dc40c18ad0f187de726962: Network error: Client connection negotiation failed: client connection to 127.3.152.193:34987: connect: Connection refused (error 111)
I20260925 10:17:08.298933 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23771
I20260925 10:17:08.303701 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23418
2026-09-25T10:17:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (758 ms)
[----------] 6 tests from EncryptionPolicyTest (8287 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:08 dist-test-slave-dzwq krb5kdc[23942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:08 dist-test-slave-dzwq krb5kdc[23942](info): set up 2 sockets
Sep 25 10:17:08 dist-test-slave-dzwq krb5kdc[23942](info): commencing operation
krb5kdc: starting...
W20260925 10:17:09.356784 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[23942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:09.481014 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:09.568404 23955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:09.568607 23955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:09.568631 23955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:09.569770 23955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:09.569808 23955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:09.569821 23955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:09.569833 23955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:09.569845 23955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:09.571238 23955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:09.571425 23955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:09.571591 23955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:09.576671 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:09.576591 23963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:09.576594 23961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:09.576942 23955 server_base.cc:1034] running on GCE node
I20260925 10:17:09.577207 23955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:09.577507 23955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:09.578676 23955 hybrid_clock.cc:648] HybridClock initialized: now 1790331429578665 us; error 31 us; skew 500 ppm
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[23942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:09.587606 23955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:09.588243 23955 webserver.cc:460] Webserver started at http://127.0.0.1:33553/ using document root <none> and password file <none>
I20260925 10:17:09.588502 23955 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:09.588577 23955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:09.588750 23955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:09.589879 23955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "75a533c5933d4ce2a08687d2591ab300"
format_stamp: "Formatted at 2026-09-25 10:17:09 on dist-test-slave-dzwq"
server_key: "585fcc8dea104534082f987ac998670c"
server_key_iv: "6d5f0a7533db2d08176f30b935952ae8"
server_key_version: "encryptionkey@0"
I20260925 10:17:09.590286 23955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "75a533c5933d4ce2a08687d2591ab300"
format_stamp: "Formatted at 2026-09-25 10:17:09 on dist-test-slave-dzwq"
server_key: "585fcc8dea104534082f987ac998670c"
server_key_iv: "6d5f0a7533db2d08176f30b935952ae8"
server_key_version: "encryptionkey@0"
I20260925 10:17:09.591743 23955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:09.592641 23970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:09.592829 23955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:09.592932 23955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "75a533c5933d4ce2a08687d2591ab300"
format_stamp: "Formatted at 2026-09-25 10:17:09 on dist-test-slave-dzwq"
server_key: "585fcc8dea104534082f987ac998670c"
server_key_iv: "6d5f0a7533db2d08176f30b935952ae8"
server_key_version: "encryptionkey@0"
I20260925 10:17:09.593032 23955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:09.606911 23955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:09.609179 23955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:09.609366 23955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:09.613992 23955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41265
I20260925 10:17:09.614017 24022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41265 every 8 connection(s)
I20260925 10:17:09.614413 23955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:09.615411 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:09.616434 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 23955
I20260925 10:17:09.616587 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:09.616786 3683 external_mini_cluster.cc:1428] Setting key 7275e6a7c03a6f1e2205b250e3b24d26
I20260925 10:17:09.618172 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300: Bootstrap starting.
I20260925 10:17:09.618781 24023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:09.619086 24023 log.cc:824] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:09.619802 24023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300: No bootstrap required, opened a new log
I20260925 10:17:09.621657 24023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260925 10:17:09.621837 24023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:09.621876 24023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a533c5933d4ce2a08687d2591ab300, State: Initialized, Role: FOLLOWER
I20260925 10:17:09.622020 24023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260925 10:17:09.622110 24023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:09.622148 24023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:09.622206 24023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:09.622831 24023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260925 10:17:09.622918 24023 leader_election.cc:302] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a533c5933d4ce2a08687d2591ab300; no voters:
I20260925 10:17:09.623123 24023 leader_election.cc:288] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:09.623169 24028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:09.623293 24028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [term 1 LEADER]: Becoming Leader. State: Replica: 75a533c5933d4ce2a08687d2591ab300, State: Running, Role: LEADER
I20260925 10:17:09.623373 24028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } }
I20260925 10:17:09.623404 24023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:09.623715 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a533c5933d4ce2a08687d2591ab300. Latest consensus state: current_term: 1 leader_uuid: "75a533c5933d4ce2a08687d2591ab300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } } }
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[23942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:17:09.623726 24029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a533c5933d4ce2a08687d2591ab300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a533c5933d4ce2a08687d2591ab300" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41265 } } }
I20260925 10:17:09.623800 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:09.623831 24029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:09.623999 24034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:09.624541 24034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:09.626157 24034 catalog_manager.cc:1347] Generated new cluster ID: 20e021cb2e34466483732e4441d36376
I20260925 10:17:09.626209 24034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:09.628495 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:09.617427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52509 (local address 127.0.0.1:41265)
0925 10:17:09.617658 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:09.617666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:09.617691 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:17:09.618135 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:09.618142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:09.618192 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:09.618362 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:09.619104 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:09.620186 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:09.620973 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:09.621394 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:09.621526 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:09.624672 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:09.624677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:09.624678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:09.624690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:09.626725 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:09.627171 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:09.627174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:09.627175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:09.627305 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:09.627542 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:09.627543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:09.627543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:09.627852 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:09.628040 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:09.628154 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 10:17:09.628259 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":145,"threads_started":1}
I20260925 10:17:09.629684 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:09.652397 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:09.644380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43601 (local address 127.0.0.1:41265)
0925 10:17:09.644437 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:09.644440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:09.644622 (+ 182us) server_negotiation.cc:407] Connection header received
0925 10:17:09.644696 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:09.644697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:09.644708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:09.644767 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:09.645597 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:09.646354 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:09.647430 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:09.647749 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:09.647810 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:09.649433 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:09.649434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:09.649434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:09.649442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:09.651034 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:09.651489 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:09.651490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:09.651490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:09.651552 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:09.651795 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:09.651796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:09.651797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:09.651922 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:09.652105 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:09.652227 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:17:09.652295 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 10:17:09.653687 24034 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:09.654446 24034 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:09.663403 24034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a533c5933d4ce2a08687d2591ab300: Generated new TSK 0
I20260925 10:17:09.663615 24034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:09.667613 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 23955
W20260925 10:17:09.671765 23940 connection.cc:535] client connection to 127.0.0.1:41265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41265): Connection reset by peer (error 104)
2026-09-25T10:17:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[24054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[24054](info): set up 2 sockets
Sep 25 10:17:09 dist-test-slave-dzwq krb5kdc[24054](info): commencing operation
krb5kdc: starting...
W20260925 10:17:10.748806 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:10 dist-test-slave-dzwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:10.871577 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:10.953987 24067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:10.954223 24067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:10.954249 24067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:10.955484 24067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:10.955528 24067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:10.955551 24067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:10.955564 24067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:10.955576 24067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:10.957325 24067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:10.957556 24067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:10.957737 24067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:10.962823 24075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:10.962849 24073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:10.962823 24072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:10.963132 24067 server_base.cc:1034] running on GCE node
I20260925 10:17:10.963418 24067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:10.963735 24067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:10.964905 24067 hybrid_clock.cc:648] HybridClock initialized: now 1790331430964875 us; error 46 us; skew 500 ppm
Sep 25 10:17:10 dist-test-slave-dzwq krb5kdc[24054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:10.967582 24067 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:10.968189 24067 webserver.cc:460] Webserver started at http://127.0.0.1:39659/ using document root <none> and password file <none>
I20260925 10:17:10.968430 24067 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:10.968498 24067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:10.968709 24067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:10.969816 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "3348723b86d144a7acc90e28bc4e615b"
format_stamp: "Formatted at 2026-09-25 10:17:10 on dist-test-slave-dzwq"
server_key: "45e9239171d78df2bb3cc54257c32076"
server_key_iv: "f485fe4663c6c923516d6830d4525b93"
server_key_version: "encryptionkey@0"
I20260925 10:17:10.970199 24067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "3348723b86d144a7acc90e28bc4e615b"
format_stamp: "Formatted at 2026-09-25 10:17:10 on dist-test-slave-dzwq"
server_key: "45e9239171d78df2bb3cc54257c32076"
server_key_iv: "f485fe4663c6c923516d6830d4525b93"
server_key_version: "encryptionkey@0"
I20260925 10:17:10.971769 24067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:10.972790 24082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:10.973006 24067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:10.973134 24067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "3348723b86d144a7acc90e28bc4e615b"
format_stamp: "Formatted at 2026-09-25 10:17:10 on dist-test-slave-dzwq"
server_key: "45e9239171d78df2bb3cc54257c32076"
server_key_iv: "f485fe4663c6c923516d6830d4525b93"
server_key_version: "encryptionkey@0"
I20260925 10:17:10.973238 24067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:10.981218 24067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:10.983306 24067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:10.983494 24067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:10.988153 24067 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38283
I20260925 10:17:10.988224 24134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38283 every 8 connection(s)
I20260925 10:17:10.988603 24067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:10.989627 24135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:10.992149 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: Bootstrap starting.
I20260925 10:17:10.992769 24135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:10.993070 24135 log.cc:824] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:10.993891 24135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: No bootstrap required, opened a new log
I20260925 10:17:10.995548 24135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260925 10:17:10.995697 24135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:10.995719 24135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3348723b86d144a7acc90e28bc4e615b, State: Initialized, Role: FOLLOWER
I20260925 10:17:10.995882 24135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260925 10:17:10.995944 24135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:10.996021 24135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:10.996075 24135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:10.996697 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24067
I20260925 10:17:10.996760 24135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260925 10:17:10.996835 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:10.996881 24135 leader_election.cc:302] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3348723b86d144a7acc90e28bc4e615b; no voters:
I20260925 10:17:10.997061 3683 external_mini_cluster.cc:1428] Setting key 6fc309bb5bfda7d89116ef687de90a5c
I20260925 10:17:10.997069 24135 leader_election.cc:288] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:10.997128 24138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:10.997306 24138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [term 1 LEADER]: Becoming Leader. State: Replica: 3348723b86d144a7acc90e28bc4e615b, State: Running, Role: LEADER
I20260925 10:17:10.997372 24135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:10.997422 24138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } }
I20260925 10:17:10.997797 24138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3348723b86d144a7acc90e28bc4e615b. Latest consensus state: current_term: 1 leader_uuid: "3348723b86d144a7acc90e28bc4e615b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } } }
I20260925 10:17:10.997880 24138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:10.998158 24139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3348723b86d144a7acc90e28bc4e615b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3348723b86d144a7acc90e28bc4e615b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38283 } } }
I20260925 10:17:10.998250 24139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b [sys.catalog]: This master's current role is: LEADER
W20260925 10:17:10.999284 24155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:10.999347 24155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:10.999425 24146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:10.999607 24146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:11.001165 24146 catalog_manager.cc:1347] Generated new cluster ID: 343ee67ead574016b07b39a0944792ea
I20260925 10:17:11.001217 24146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:11.004005 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:10.997774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45877 (local address 127.0.0.1:38283)
0925 10:17:10.997997 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:10.998009 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:10.998062 (+ 53us) server_negotiation.cc:407] Connection header received
0925 10:17:10.998204 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:10.998215 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:10.998282 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:10.998433 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:10.998721 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:11.000297 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:11.001452 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:11.002078 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:11.002550 (+ 472us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:11.003063 (+ 513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:11.003068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:11.003068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:11.003299 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:11.003339 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:11.003355 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:11.003666 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:11.003710 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 10:17:11.003739 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260925 10:17:11.004877 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:11.007360 24146 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:11.007849 24146 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:11.010399 24144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:11.006415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53771 (local address 127.0.0.1:38283)
0925 10:17:11.006448 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:11.006451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:11.006601 (+ 150us) server_negotiation.cc:407] Connection header received
0925 10:17:11.006696 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:11.006697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:11.006708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:11.006738 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:11.006957 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:11.007777 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:11.008861 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:11.009199 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:11.009241 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:11.010048 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:11.010049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:11.010049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:11.010055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:11.010069 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:11.010074 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:11.010106 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:11.010252 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 10:17:11.010266 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 10:17:11.018887 24146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3348723b86d144a7acc90e28bc4e615b: Generated new TSK 0
I20260925 10:17:11.019132 24146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:11.027647 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24067
W20260925 10:17:11.032269 24052 connection.cc:535] client connection to 127.0.0.1:38283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38283): Connection reset by peer (error 104)
2026-09-25T10:17:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:11 dist-test-slave-dzwq krb5kdc[24166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:11 dist-test-slave-dzwq krb5kdc[24166](info): set up 2 sockets
Sep 25 10:17:11 dist-test-slave-dzwq krb5kdc[24166](info): commencing operation
krb5kdc: starting...
W20260925 10:17:12.068804 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:12 dist-test-slave-dzwq krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:12.191171 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40095
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:12.272532 24179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:12.272768 24179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:12.272804 24179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:12.273959 24179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:12.274042 24179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:12.274070 24179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:12.274101 24179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:12.274122 24179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:12.275563 24179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:12.275830 24179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:12.276079 24179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:12.280920 24185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:12.280951 24184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:12.280951 24187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:12.281354 24179 server_base.cc:1034] running on GCE node
I20260925 10:17:12.281580 24179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:12.281826 24179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:12.282961 24179 hybrid_clock.cc:648] HybridClock initialized: now 1790331432282936 us; error 40 us; skew 500 ppm
Sep 25 10:17:12 dist-test-slave-dzwq krb5kdc[24166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:12.285491 24179 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:12.285964 24179 webserver.cc:460] Webserver started at http://127.0.0.1:42761/ using document root <none> and password file <none>
I20260925 10:17:12.286182 24179 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:12.286226 24179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:12.286311 24179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:12.287302 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "f76a1178905a41609eafd656c6bf59ac"
format_stamp: "Formatted at 2026-09-25 10:17:12 on dist-test-slave-dzwq"
server_key: "0f8a8b1efd0a26a9550c455b7d5b8c19"
server_key_iv: "2ab64f7c16dbcb8fbfa2a733bb68fa0b"
server_key_version: "encryptionkey@0"
I20260925 10:17:12.287616 24179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "f76a1178905a41609eafd656c6bf59ac"
format_stamp: "Formatted at 2026-09-25 10:17:12 on dist-test-slave-dzwq"
server_key: "0f8a8b1efd0a26a9550c455b7d5b8c19"
server_key_iv: "2ab64f7c16dbcb8fbfa2a733bb68fa0b"
server_key_version: "encryptionkey@0"
I20260925 10:17:12.288996 24179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:12.289731 24194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:12.289918 24179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:12.289985 24179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "f76a1178905a41609eafd656c6bf59ac"
format_stamp: "Formatted at 2026-09-25 10:17:12 on dist-test-slave-dzwq"
server_key: "0f8a8b1efd0a26a9550c455b7d5b8c19"
server_key_iv: "2ab64f7c16dbcb8fbfa2a733bb68fa0b"
server_key_version: "encryptionkey@0"
I20260925 10:17:12.290036 24179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:12.293351 24179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:12.295568 24179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:12.295677 24179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:12.299678 24179 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40095
I20260925 10:17:12.299711 24246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40095 every 8 connection(s)
I20260925 10:17:12.300114 24179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:12.301082 24247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:12.303318 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac: Bootstrap starting.
I20260925 10:17:12.303866 24247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:12.304190 24247 log.cc:824] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:12.304884 24247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac: No bootstrap required, opened a new log
I20260925 10:17:12.306140 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24179
I20260925 10:17:12.306249 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:12.306447 3683 external_mini_cluster.cc:1428] Setting key 25a0a134d7200c837f266f715771a633
I20260925 10:17:12.306643 24247 raft_consensus.cc:348] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } }
I20260925 10:17:12.306852 24247 raft_consensus.cc:374] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:12.306937 24247 raft_consensus.cc:729] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76a1178905a41609eafd656c6bf59ac, State: Initialized, Role: FOLLOWER
I20260925 10:17:12.307104 24247 consensus_queue.cc:260] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } }
I20260925 10:17:12.307202 24247 raft_consensus.cc:388] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:12.307240 24247 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:12.307294 24247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:12.308359 24247 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } }
I20260925 10:17:12.308488 24247 leader_election.cc:302] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f76a1178905a41609eafd656c6bf59ac; no voters:
I20260925 10:17:12.308665 24247 leader_election.cc:288] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:12.308678 24252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:12.308894 24252 raft_consensus.cc:686] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [term 1 LEADER]: Becoming Leader. State: Replica: f76a1178905a41609eafd656c6bf59ac, State: Running, Role: LEADER
I20260925 10:17:12.308969 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:12.307110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37775 (local address 127.0.0.1:40095)
0925 10:17:12.307353 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:12.307363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:12.307397 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:17:12.307508 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:12.307513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:12.307537 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:12.307684 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:12.308458 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:12.308462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:12.308462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:12.308466 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:12.308514 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:12.308530 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:12.308604 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:12.308701 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 10:17:12.308747 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260925 10:17:12.309018 24247 sys_catalog.cc:564] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:12.309018 24252 consensus_queue.cc:237] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } }
I20260925 10:17:12.310092 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:12.310114 24253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f76a1178905a41609eafd656c6bf59ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } } }
I20260925 10:17:12.310223 24253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:12.310434 24263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:12.310461 24254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f76a1178905a41609eafd656c6bf59ac. Latest consensus state: current_term: 1 leader_uuid: "f76a1178905a41609eafd656c6bf59ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76a1178905a41609eafd656c6bf59ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40095 } } }
I20260925 10:17:12.310710 24254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:12.311038 24263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:12.312984 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:12.311654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58177 (local address 127.0.0.1:40095)
0925 10:17:12.311692 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:12.311694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:12.311841 (+ 147us) server_negotiation.cc:407] Connection header received
0925 10:17:12.311923 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:12.311924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:12.311930 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:12.312047 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:12.312746 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:12.312748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:12.312748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:12.312752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:12.312769 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:12.312773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:12.312836 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:12.312909 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:17:12.312922 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:17:12.313020 24263 catalog_manager.cc:1347] Generated new cluster ID: bdb4c18c97344631b4c0b7f39e84b42c
I20260925 10:17:12.313063 24263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:12.330322 24263 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:12.330554 24263 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:12.339669 24263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f76a1178905a41609eafd656c6bf59ac: Generated new TSK 0
I20260925 10:17:12.340001 24263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:12.352088 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24179
2026-09-25T10:17:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:12 dist-test-slave-dzwq krb5kdc[24278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:12 dist-test-slave-dzwq krb5kdc[24278](info): set up 2 sockets
Sep 25 10:17:12 dist-test-slave-dzwq krb5kdc[24278](info): commencing operation
krb5kdc: starting...
W20260925 10:17:13.396803 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:13.519446 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:13.609472 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:13.609689 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:13.609711 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:13.610821 24291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:13.610857 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:13.610872 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:13.610884 24291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:13.610896 24291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:13.612367 24291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:13.612612 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:13.612920 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:13.617887 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:13.617888 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:13.617930 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:13.618232 24291 server_base.cc:1034] running on GCE node
I20260925 10:17:13.618498 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:13.618794 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:13.619947 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790331433619931 us; error 29 us; skew 500 ppm
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:13.622480 24291 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:13.622952 24291 webserver.cc:460] Webserver started at http://127.0.0.1:33823/ using document root <none> and password file <none>
I20260925 10:17:13.623195 24291 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:13.623265 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:13.623420 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:13.624672 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "12718eab02294573936ff4be0790422f"
format_stamp: "Formatted at 2026-09-25 10:17:13 on dist-test-slave-dzwq"
server_key: "637b9b2ca6838c04907019c088875ab8"
server_key_iv: "976dbe3136c1edaf758fd235061095bb"
server_key_version: "encryptionkey@0"
I20260925 10:17:13.625047 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "12718eab02294573936ff4be0790422f"
format_stamp: "Formatted at 2026-09-25 10:17:13 on dist-test-slave-dzwq"
server_key: "637b9b2ca6838c04907019c088875ab8"
server_key_iv: "976dbe3136c1edaf758fd235061095bb"
server_key_version: "encryptionkey@0"
I20260925 10:17:13.626405 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:13.627163 24306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:13.627357 24291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:13.627460 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "12718eab02294573936ff4be0790422f"
format_stamp: "Formatted at 2026-09-25 10:17:13 on dist-test-slave-dzwq"
server_key: "637b9b2ca6838c04907019c088875ab8"
server_key_iv: "976dbe3136c1edaf758fd235061095bb"
server_key_version: "encryptionkey@0"
I20260925 10:17:13.627542 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:13.648831 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:13.650889 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:13.651067 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:13.655246 24291 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36005
I20260925 10:17:13.655284 24358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36005 every 8 connection(s)
I20260925 10:17:13.655675 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:13.656770 24359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:13.659076 24359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f: Bootstrap starting.
I20260925 10:17:13.659610 24359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:13.659843 24359 log.cc:824] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:13.660604 24359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f: No bootstrap required, opened a new log
I20260925 10:17:13.662127 24359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } }
I20260925 10:17:13.662262 24359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:13.662284 24359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12718eab02294573936ff4be0790422f, State: Initialized, Role: FOLLOWER
I20260925 10:17:13.662425 24359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } }
I20260925 10:17:13.662515 24359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:13.662539 24359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:13.662568 24359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:13.663046 24359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } }
I20260925 10:17:13.663126 24359 leader_election.cc:302] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12718eab02294573936ff4be0790422f; no voters:
I20260925 10:17:13.663249 24359 leader_election.cc:288] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:13.663312 24362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:13.663491 24362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [term 1 LEADER]: Becoming Leader. State: Replica: 12718eab02294573936ff4be0790422f, State: Running, Role: LEADER
I20260925 10:17:13.663570 24359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:13.663622 24362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } }
I20260925 10:17:13.664042 24364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12718eab02294573936ff4be0790422f. Latest consensus state: current_term: 1 leader_uuid: "12718eab02294573936ff4be0790422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } } }
I20260925 10:17:13.664126 24364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:13.664191 24363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12718eab02294573936ff4be0790422f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12718eab02294573936ff4be0790422f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36005 } } }
I20260925 10:17:13.664247 24363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:13.664413 24370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:13.664896 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24291
I20260925 10:17:13.665015 24370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:13.665025 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:13.665257 3683 external_mini_cluster.cc:1428] Setting key 4951b1068ca9a62eba5a33eaa2ad7092
I20260925 10:17:13.666929 24370 catalog_manager.cc:1347] Generated new cluster ID: 55bad8d1896d48d6a565f6740991f2fb
I20260925 10:17:13.667016 24370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:17:13.677083 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:13.665917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59217 (local address 127.0.0.1:36005)
0925 10:17:13.666066 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:13.666075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:13.666104 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:17:13.666229 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:13.666235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:13.666291 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:13.666425 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:13.667237 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:13.668710 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:13.669742 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:13.670356 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:13.670467 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:13.673290 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:13.673294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:13.673295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:13.673306 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:13.675247 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:13.675692 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:13.675695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:13.675695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:13.675797 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:13.676040 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:13.676041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:13.676041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:13.676344 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:13.676482 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:13.676612 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 10:17:13.676847 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260925 10:17:13.678001 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:13.683640 24370 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:13.684340 24370 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:13.686673 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:13.679438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44671 (local address 127.0.0.1:36005)
0925 10:17:13.679475 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:13.679477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:13.679644 (+ 167us) server_negotiation.cc:407] Connection header received
0925 10:17:13.679692 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:13.679693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:13.679703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:13.679754 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:13.680401 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:13.681140 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:13.682159 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:13.682466 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:13.682531 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:13.683922 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:13.683923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:13.683924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:13.683930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:13.685538 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:13.685922 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:13.685923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:13.685923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:13.685975 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:13.686181 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:13.686182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:13.686182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:13.686289 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:13.686374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:13.686496 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 10:17:13.686578 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:17:13.688354 24370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12718eab02294573936ff4be0790422f: Generated new TSK 0
I20260925 10:17:13.688542 24370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:13.692312 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24291
2026-09-25T10:17:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24390](info): set up 2 sockets
Sep 25 10:17:13 dist-test-slave-dzwq krb5kdc[24390](info): commencing operation
krb5kdc: starting...
W20260925 10:17:14.732798 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:14 dist-test-slave-dzwq krb5kdc[24390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:14.859794 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:14.947996 24403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:14.948208 24403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:14.948232 24403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:14.949325 24403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:14.949361 24403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:14.949375 24403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:14.949388 24403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:14.949399 24403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:14.950809 24403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:14.951004 24403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:14.951177 24403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:14.956281 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:14.956295 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:14.956295 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:14.956837 24403 server_base.cc:1034] running on GCE node
I20260925 10:17:14.957077 24403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:14.957362 24403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:14.958544 24403 hybrid_clock.cc:648] HybridClock initialized: now 1790331434958521 us; error 36 us; skew 500 ppm
Sep 25 10:17:14 dist-test-slave-dzwq krb5kdc[24390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:14.966096 24403 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:14.966616 24403 webserver.cc:460] Webserver started at http://127.0.0.1:36115/ using document root <none> and password file <none>
I20260925 10:17:14.966876 24403 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:14.966948 24403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:14.967113 24403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:14.968297 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "8b864901dad944ae9ef8c9ac6063ed0f"
format_stamp: "Formatted at 2026-09-25 10:17:14 on dist-test-slave-dzwq"
server_key: "44b8b0669a8774f756503f4646c58888"
server_key_iv: "9e3339e99c53dbae84a2b4dcf6cced1f"
server_key_version: "encryptionkey@0"
I20260925 10:17:14.968659 24403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "8b864901dad944ae9ef8c9ac6063ed0f"
format_stamp: "Formatted at 2026-09-25 10:17:14 on dist-test-slave-dzwq"
server_key: "44b8b0669a8774f756503f4646c58888"
server_key_iv: "9e3339e99c53dbae84a2b4dcf6cced1f"
server_key_version: "encryptionkey@0"
I20260925 10:17:14.969990 24403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:14.970745 24418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:14.970927 24403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:14.971035 24403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8b864901dad944ae9ef8c9ac6063ed0f"
format_stamp: "Formatted at 2026-09-25 10:17:14 on dist-test-slave-dzwq"
server_key: "44b8b0669a8774f756503f4646c58888"
server_key_iv: "9e3339e99c53dbae84a2b4dcf6cced1f"
server_key_version: "encryptionkey@0"
I20260925 10:17:14.971122 24403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:14.988744 24403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:14.990729 24403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:14.990898 24403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:14.995373 24403 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35571
I20260925 10:17:14.995441 24470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35571 every 8 connection(s)
I20260925 10:17:14.995810 24403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:14.996865 24471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:14.999233 24471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: Bootstrap starting.
I20260925 10:17:14.999846 24471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:15.000192 24471 log.cc:824] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:15.000917 24471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: No bootstrap required, opened a new log
I20260925 10:17:15.002627 24471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260925 10:17:15.002767 24471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:15.002791 24471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b864901dad944ae9ef8c9ac6063ed0f, State: Initialized, Role: FOLLOWER
I20260925 10:17:15.002951 24471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260925 10:17:15.003017 24471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:15.003039 24471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:15.003067 24471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:15.003625 24471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260925 10:17:15.003717 24471 leader_election.cc:302] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b864901dad944ae9ef8c9ac6063ed0f; no voters:
I20260925 10:17:15.003934 24471 leader_election.cc:288] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:15.004092 24474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:15.004251 24471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:15.004278 24474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [term 1 LEADER]: Becoming Leader. State: Replica: 8b864901dad944ae9ef8c9ac6063ed0f, State: Running, Role: LEADER
I20260925 10:17:15.004472 24474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } }
I20260925 10:17:15.005098 24476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b864901dad944ae9ef8c9ac6063ed0f. Latest consensus state: current_term: 1 leader_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } } }
I20260925 10:17:15.005184 24476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:15.005275 24475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b864901dad944ae9ef8c9ac6063ed0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35571 } } }
I20260925 10:17:15.005357 24475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:15.005553 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24403
I20260925 10:17:15.005683 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:15.005887 3683 external_mini_cluster.cc:1428] Setting key 6e929a4cb0ad5edd7c7a156c6cefa2a2
W20260925 10:17:15.006188 24490 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:15.006291 24490 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:15.006381 24483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:15.006633 24483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:15.008724 24483 catalog_manager.cc:1347] Generated new cluster ID: b12ef49fde3748988e8e0004ff730a95
I20260925 10:17:15.008785 24483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:15.011919 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:15.006551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58805 (local address 127.0.0.1:35571)
0925 10:17:15.006722 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:15.006731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:15.006762 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:17:15.006881 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:15.006888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:15.006949 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:15.007107 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:15.007335 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:15.008567 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:15.009726 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:15.010333 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:15.010450 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:15.011207 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:15.011211 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:15.011211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:15.011429 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:15.011470 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:15.011485 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:15.011551 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:15.011649 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 10:17:15.011709 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260925 10:17:15.013041 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:15.015504 24483 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:15.016146 24483 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:15.022135 24483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b864901dad944ae9ef8c9ac6063ed0f: Generated new TSK 0
I20260925 10:17:15.022338 24483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:15.023787 24493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:15.014602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56677 (local address 127.0.0.1:35571)
0925 10:17:15.014638 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:15.014640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:15.020199 (+ 5559us) server_negotiation.cc:407] Connection header received
0925 10:17:15.020267 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:15.020268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:15.020280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:15.020346 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:15.020500 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:15.021292 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:15.022467 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:15.022717 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:15.022774 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:15.023448 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:15.023449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:15.023449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:15.023454 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:15.023467 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:15.023471 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:15.023595 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:15.023675 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 10:17:15.023700 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:15.025600 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24403
2026-09-25T10:17:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:15 dist-test-slave-dzwq krb5kdc[24502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:15 dist-test-slave-dzwq krb5kdc[24502](info): set up 2 sockets
Sep 25 10:17:15 dist-test-slave-dzwq krb5kdc[24502](info): commencing operation
krb5kdc: starting...
W20260925 10:17:16.056825 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:16 dist-test-slave-dzwq krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:16.179827 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:16.261715 24515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:16.261978 24515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:16.262022 24515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:16.263213 24515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:16.263286 24515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:16.263311 24515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:16.263340 24515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:16.263370 24515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:16.264979 24515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:16.265264 24515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:16.265501 24515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:16.270721 24521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:16.270725 24523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:16.270725 24520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:16.270805 24515 server_base.cc:1034] running on GCE node
I20260925 10:17:16.271211 24515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:16.271522 24515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:16.272691 24515 hybrid_clock.cc:648] HybridClock initialized: now 1790331436272677 us; error 45 us; skew 500 ppm
Sep 25 10:17:16 dist-test-slave-dzwq krb5kdc[24502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:16.275200 24515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:16.275700 24515 webserver.cc:460] Webserver started at http://127.0.0.1:41237/ using document root <none> and password file <none>
I20260925 10:17:16.275939 24515 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:16.276046 24515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:16.276211 24515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:16.277316 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "ef00f9f7ee8848219064008129196d4e"
format_stamp: "Formatted at 2026-09-25 10:17:16 on dist-test-slave-dzwq"
server_key: "682859cb826a446bd18bee52f61b18c6"
server_key_iv: "74351613468dfffdd7a945de61b1698a"
server_key_version: "encryptionkey@0"
I20260925 10:17:16.277693 24515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "ef00f9f7ee8848219064008129196d4e"
format_stamp: "Formatted at 2026-09-25 10:17:16 on dist-test-slave-dzwq"
server_key: "682859cb826a446bd18bee52f61b18c6"
server_key_iv: "74351613468dfffdd7a945de61b1698a"
server_key_version: "encryptionkey@0"
I20260925 10:17:16.279065 24515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:16.279898 24530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:16.280108 24515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:16.280213 24515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "ef00f9f7ee8848219064008129196d4e"
format_stamp: "Formatted at 2026-09-25 10:17:16 on dist-test-slave-dzwq"
server_key: "682859cb826a446bd18bee52f61b18c6"
server_key_iv: "74351613468dfffdd7a945de61b1698a"
server_key_version: "encryptionkey@0"
I20260925 10:17:16.280299 24515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:16.283764 24515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:16.285841 24515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:16.286005 24515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:16.290206 24515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42179
I20260925 10:17:16.290244 24582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42179 every 8 connection(s)
I20260925 10:17:16.290609 24515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:16.291663 24583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:16.293983 24583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e: Bootstrap starting.
I20260925 10:17:16.294521 24583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:16.294765 24583 log.cc:824] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:16.295013 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24515
I20260925 10:17:16.295183 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:16.295384 3683 external_mini_cluster.cc:1428] Setting key 420273e1a8406e41fba1c478dc3132ec
I20260925 10:17:16.295627 24583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e: No bootstrap required, opened a new log
I20260925 10:17:16.297487 24583 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } }
I20260925 10:17:16.297657 24583 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:16.297724 24583 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef00f9f7ee8848219064008129196d4e, State: Initialized, Role: FOLLOWER
I20260925 10:17:16.297870 24583 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } }
I20260925 10:17:16.297979 24583 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:16.298012 24583 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:16.298072 24583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:16.298892 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:16.295901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35919 (local address 127.0.0.1:42179)
0925 10:17:16.296090 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:16.296099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:16.296130 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:17:16.297470 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:16.297476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:16.297504 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:16.297670 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:16.298368 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:16.298371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:16.298372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:16.298376 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:16.298436 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:16.298448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:16.298527 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:16.298598 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 10:17:16.298743 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260925 10:17:16.299007 24583 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } }
I20260925 10:17:16.299132 24583 leader_election.cc:302] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef00f9f7ee8848219064008129196d4e; no voters:
I20260925 10:17:16.299301 24583 leader_election.cc:288] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:16.299361 24588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:16.299496 24588 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [term 1 LEADER]: Becoming Leader. State: Replica: ef00f9f7ee8848219064008129196d4e, State: Running, Role: LEADER
I20260925 10:17:16.299538 24583 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:16.299620 24588 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } }
I20260925 10:17:16.300010 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:16.300213 24588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef00f9f7ee8848219064008129196d4e. Latest consensus state: current_term: 1 leader_uuid: "ef00f9f7ee8848219064008129196d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } } }
I20260925 10:17:16.300290 24588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:16.300227 24589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef00f9f7ee8848219064008129196d4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef00f9f7ee8848219064008129196d4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42179 } } }
I20260925 10:17:16.300341 24589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:16.301224 24597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:16.302126 24597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:16.302807 24587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:16.301582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35681 (local address 127.0.0.1:42179)
0925 10:17:16.301620 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:16.301622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:16.301713 (+ 91us) server_negotiation.cc:407] Connection header received
0925 10:17:16.301819 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:16.301820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:16.301826 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:16.301882 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:16.302576 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:16.302577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:16.302577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:16.302581 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:16.302596 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:16.302600 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:16.302649 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:16.302707 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 10:17:16.302725 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:17:16.303822 24597 catalog_manager.cc:1347] Generated new cluster ID: 0f8aa64474f44f6d83cf32f13e69f1ed
I20260925 10:17:16.303877 24597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:16.312686 24597 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:16.312870 24597 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:16.318121 24597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef00f9f7ee8848219064008129196d4e: Generated new TSK 0
I20260925 10:17:16.318328 24597 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:16.323976 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24515
2026-09-25T10:17:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:16 dist-test-slave-dzwq krb5kdc[24614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:16 dist-test-slave-dzwq krb5kdc[24614](info): set up 2 sockets
Sep 25 10:17:16 dist-test-slave-dzwq krb5kdc[24614](info): commencing operation
krb5kdc: starting...
W20260925 10:17:17.356949 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:17.478855 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:17.564787 24627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:17.565037 24627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:17.565076 24627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:17.566296 24627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:17.566375 24627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:17.566401 24627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:17.566432 24627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:17.566459 24627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:17.568053 24627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:17.568351 24627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:17.568593 24627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:17.573550 24633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:17.573545 24635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:17.573541 24632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:17.574146 24627 server_base.cc:1034] running on GCE node
I20260925 10:17:17.574363 24627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:17.574676 24627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:17.575882 24627 hybrid_clock.cc:648] HybridClock initialized: now 1790331437575866 us; error 30 us; skew 500 ppm
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:17.578785 24627 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:17.579255 24627 webserver.cc:460] Webserver started at http://127.0.0.1:38595/ using document root <none> and password file <none>
I20260925 10:17:17.579488 24627 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:17.579536 24627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:17.579711 24627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:17.580906 24627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "8d0ad992998b47e58f58de122d1979cd"
format_stamp: "Formatted at 2026-09-25 10:17:17 on dist-test-slave-dzwq"
server_key: "e5854cd975d29e74fe812cb79cba9709"
server_key_iv: "30fb1f3a858841ea41206dc87ad7233a"
server_key_version: "encryptionkey@0"
I20260925 10:17:17.581302 24627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "8d0ad992998b47e58f58de122d1979cd"
format_stamp: "Formatted at 2026-09-25 10:17:17 on dist-test-slave-dzwq"
server_key: "e5854cd975d29e74fe812cb79cba9709"
server_key_iv: "30fb1f3a858841ea41206dc87ad7233a"
server_key_version: "encryptionkey@0"
I20260925 10:17:17.582659 24627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 10:17:17.583423 24642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:17.583691 24627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:17.583774 24627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8d0ad992998b47e58f58de122d1979cd"
format_stamp: "Formatted at 2026-09-25 10:17:17 on dist-test-slave-dzwq"
server_key: "e5854cd975d29e74fe812cb79cba9709"
server_key_iv: "30fb1f3a858841ea41206dc87ad7233a"
server_key_version: "encryptionkey@0"
I20260925 10:17:17.583830 24627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:17.619426 24627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:17.621454 24627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:17.621568 24627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:17.625694 24627 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40131
I20260925 10:17:17.625725 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40131 every 8 connection(s)
I20260925 10:17:17.626092 24627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:17.627063 24695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:17.629413 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd: Bootstrap starting.
I20260925 10:17:17.630009 24695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:17.630308 24695 log.cc:824] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:17.630999 24695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd: No bootstrap required, opened a new log
I20260925 10:17:17.632707 24695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } }
I20260925 10:17:17.632871 24695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:17.632916 24695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0ad992998b47e58f58de122d1979cd, State: Initialized, Role: FOLLOWER
I20260925 10:17:17.633093 24695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } }
I20260925 10:17:17.633185 24695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:17.633230 24695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:17.633283 24695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:17.633821 24695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } }
I20260925 10:17:17.633940 24695 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0ad992998b47e58f58de122d1979cd; no voters:
I20260925 10:17:17.634057 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24627
I20260925 10:17:17.634161 24695 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:17.634210 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:17.634207 24698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:17.634333 24698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [term 1 LEADER]: Becoming Leader. State: Replica: 8d0ad992998b47e58f58de122d1979cd, State: Running, Role: LEADER
I20260925 10:17:17.634418 24698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } }
I20260925 10:17:17.634447 3683 external_mini_cluster.cc:1428] Setting key cfaf66f35ff8b45ed4ab069db690bd23
I20260925 10:17:17.634469 24695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:17.634932 24700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0ad992998b47e58f58de122d1979cd. Latest consensus state: current_term: 1 leader_uuid: "8d0ad992998b47e58f58de122d1979cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } } }
I20260925 10:17:17.635015 24700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:17.635192 24699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0ad992998b47e58f58de122d1979cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ad992998b47e58f58de122d1979cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40131 } } }
I20260925 10:17:17.635253 24699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:17.635387 24707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:17.635936 24707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:17.637753 24707 catalog_manager.cc:1347] Generated new cluster ID: b795d1eb51204fd79fa186e97ae4d279
I20260925 10:17:17.637804 24707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 10:17:17.644764 24707 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:17.645301 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:17.635078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:54499 (local address 127.0.0.1:40131)
0925 10:17:17.635548 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:17.635557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:17.635588 (+ 31us) server_negotiation.cc:407] Connection header received
0925 10:17:17.635711 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:17.635720 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:17.635780 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:17.635921 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:17.636797 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:17.637956 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:17.638731 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:17.639164 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:17.639282 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:17.641749 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:17.641754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:17.641754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:17.641767 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:17.643482 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:17.643884 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:17.643887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:17.643888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:17.644043 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:17.644286 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:17.644288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:17.644288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:17.644510 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:17.644690 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:17.644866 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 10:17:17.645079 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":86,"threads_started":1}
I20260925 10:17:17.645483 24707 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:17.646678 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:17.650645 24707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0ad992998b47e58f58de122d1979cd: Generated new TSK 0
I20260925 10:17:17.650830 24707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:17.655254 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:17.648270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:48009 (local address 127.0.0.1:40131)
0925 10:17:17.648307 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:17.648309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:17.648333 (+ 24us) server_negotiation.cc:407] Connection header received
0925 10:17:17.648406 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:17.648406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:17.648416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:17.648474 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:17.649104 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:17.649890 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:17.650941 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:17.651294 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:17.651365 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:17.652657 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:17.652658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:17.652659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:17.652665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:17.654169 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:17.654511 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:17.654512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:17.654512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:17.654569 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:17.654777 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:17.654778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:17.654778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:17.654903 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:17.655017 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:17.655099 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:17:17.655159 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:17:17.657150 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24627
W20260925 10:17:17.661114 24612 connection.cc:535] client connection to 127.0.0.1:40131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40131): Connection reset by peer (error 104)
2026-09-25T10:17:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24726](info): set up 2 sockets
Sep 25 10:17:17 dist-test-slave-dzwq krb5kdc[24726](info): commencing operation
krb5kdc: starting...
W20260925 10:17:18.700850 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:18 dist-test-slave-dzwq krb5kdc[24726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:18.823587 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:18.909031 24739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:18.909344 24739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:18.909399 24739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:18.910975 24739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:18.911024 24739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:18.911038 24739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:18.911051 24739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:18.911064 24739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:18.912626 24739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:18.912837 24739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:18.913019 24739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:18.917877 24747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:18.917877 24745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:18.917904 24744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:18.918216 24739 server_base.cc:1034] running on GCE node
I20260925 10:17:18.918478 24739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:18.918732 24739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:18.919881 24739 hybrid_clock.cc:648] HybridClock initialized: now 1790331438919865 us; error 29 us; skew 500 ppm
Sep 25 10:17:18 dist-test-slave-dzwq krb5kdc[24726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:18.922430 24739 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:18.922899 24739 webserver.cc:460] Webserver started at http://127.0.0.1:43661/ using document root <none> and password file <none>
I20260925 10:17:18.923092 24739 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:18.923133 24739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:18.923233 24739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:18.924355 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "24170e6f2426490a94c7797095d6c157"
format_stamp: "Formatted at 2026-09-25 10:17:18 on dist-test-slave-dzwq"
server_key: "f320a669b3c2038459560ec09688a08f"
server_key_iv: "b4de33ac1a6ea7b7504ae3c4fd7583a2"
server_key_version: "encryptionkey@0"
I20260925 10:17:18.924666 24739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "24170e6f2426490a94c7797095d6c157"
format_stamp: "Formatted at 2026-09-25 10:17:18 on dist-test-slave-dzwq"
server_key: "f320a669b3c2038459560ec09688a08f"
server_key_iv: "b4de33ac1a6ea7b7504ae3c4fd7583a2"
server_key_version: "encryptionkey@0"
I20260925 10:17:18.926038 24739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:18.926750 24754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:18.926947 24739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:18.927032 24739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "24170e6f2426490a94c7797095d6c157"
format_stamp: "Formatted at 2026-09-25 10:17:18 on dist-test-slave-dzwq"
server_key: "f320a669b3c2038459560ec09688a08f"
server_key_iv: "b4de33ac1a6ea7b7504ae3c4fd7583a2"
server_key_version: "encryptionkey@0"
I20260925 10:17:18.927089 24739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:18.970604 24739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:18.972821 24739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:18.972949 24739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:18.977303 24739 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40377
I20260925 10:17:18.977363 24806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40377 every 8 connection(s)
I20260925 10:17:18.977761 24739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:18.978842 24807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:18.979363 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24739
I20260925 10:17:18.979458 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:18.979704 3683 external_mini_cluster.cc:1428] Setting key d90a8c4399e829ae737c24eabca28aa5
I20260925 10:17:18.981812 24807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157: Bootstrap starting.
I20260925 10:17:18.982471 24807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:18.982800 24807 log.cc:824] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:18.983554 24807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157: No bootstrap required, opened a new log
I20260925 10:17:18.985354 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:18.980307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:38929 (local address 127.0.0.1:40377)
0925 10:17:18.980471 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:18.980480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:18.980507 (+ 27us) server_negotiation.cc:407] Connection header received
0925 10:17:18.980909 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:18.980916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:18.980964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:18.981082 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:18.981354 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:18.982310 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:18.983403 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:18.983847 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:18.983928 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:18.984712 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:18.984716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:18.984716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:18.984872 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:18.984900 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:18.984911 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:18.984977 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:18.985093 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 10:17:18.985136 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260925 10:17:18.986068 24807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260925 10:17:18.986261 24807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:18.986310 24807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24170e6f2426490a94c7797095d6c157, State: Initialized, Role: FOLLOWER
I20260925 10:17:18.986436 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:18.986433 24807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260925 10:17:18.986524 24807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:18.986567 24807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:18.986615 24807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:18.987242 24807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260925 10:17:18.987382 24807 leader_election.cc:302] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24170e6f2426490a94c7797095d6c157; no voters:
I20260925 10:17:18.987560 24807 leader_election.cc:288] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:18.987586 24816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:18.987790 24816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [term 1 LEADER]: Becoming Leader. State: Replica: 24170e6f2426490a94c7797095d6c157, State: Running, Role: LEADER
I20260925 10:17:18.987933 24816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } }
I20260925 10:17:18.987946 24807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:18.988387 24818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24170e6f2426490a94c7797095d6c157. Latest consensus state: current_term: 1 leader_uuid: "24170e6f2426490a94c7797095d6c157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } } }
I20260925 10:17:18.988466 24817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24170e6f2426490a94c7797095d6c157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24170e6f2426490a94c7797095d6c157" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40377 } } }
I20260925 10:17:18.988528 24818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:18.988976 24817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:18.989006 24826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:18.989528 24826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:18.991055 24826 catalog_manager.cc:1347] Generated new cluster ID: eabc3bb0742c4af199ada0eefb4fc44d
I20260925 10:17:18.991109 24826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:18.992008 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:18.987810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:54441 (local address 127.0.0.1:40377)
0925 10:17:18.987848 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:18.987851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:18.988239 (+ 388us) server_negotiation.cc:407] Connection header received
0925 10:17:18.988330 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:18.988331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:18.988342 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:18.988410 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:18.989097 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:18.989817 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:18.990579 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:18.990839 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:18.990901 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:18.991708 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:18.991709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:18.991710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:18.991714 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:18.991728 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:18.991732 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:18.991775 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:18.991904 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:17:18.991914 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:17:19.010833 24826 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:19.011642 24826 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:19.015722 24826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24170e6f2426490a94c7797095d6c157: Generated new TSK 0
I20260925 10:17:19.015918 24826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:19.021382 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24739
W20260925 10:17:19.026176 24724 connection.cc:535] client connection to 127.0.0.1:40377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40377): Connection reset by peer (error 104)
2026-09-25T10:17:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:19 dist-test-slave-dzwq krb5kdc[24838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:19 dist-test-slave-dzwq krb5kdc[24838](info): set up 2 sockets
Sep 25 10:17:19 dist-test-slave-dzwq krb5kdc[24838](info): commencing operation
krb5kdc: starting...
W20260925 10:17:20.092845 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:20 dist-test-slave-dzwq krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:20.219303 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36863
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:20.300595 24851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:20.300892 24851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:20.300936 24851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:20.302126 24851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:20.302220 24851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:20.302246 24851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:20.302276 24851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:20.302304 24851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:20.303821 24851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:20.304174 24851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:20.304430 24851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:20.309473 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:20.309481 24859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:20.309481 24857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:20.309640 24851 server_base.cc:1034] running on GCE node
I20260925 10:17:20.309939 24851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:20.310187 24851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:20.311380 24851 hybrid_clock.cc:648] HybridClock initialized: now 1790331440311326 us; error 79 us; skew 500 ppm
Sep 25 10:17:20 dist-test-slave-dzwq krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:20.320065 24851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:20.320608 24851 webserver.cc:460] Webserver started at http://127.0.0.1:38773/ using document root <none> and password file <none>
I20260925 10:17:20.320881 24851 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:20.320955 24851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:20.321110 24851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:20.322269 24851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "e955499aef4748a5962b56066e93bc71"
format_stamp: "Formatted at 2026-09-25 10:17:20 on dist-test-slave-dzwq"
server_key: "ab220b310c6f54c1039f6cfe9063ac3b"
server_key_iv: "e5945c0560d677887e1d2668deec788f"
server_key_version: "encryptionkey@0"
I20260925 10:17:20.322669 24851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "e955499aef4748a5962b56066e93bc71"
format_stamp: "Formatted at 2026-09-25 10:17:20 on dist-test-slave-dzwq"
server_key: "ab220b310c6f54c1039f6cfe9063ac3b"
server_key_iv: "e5945c0560d677887e1d2668deec788f"
server_key_version: "encryptionkey@0"
I20260925 10:17:20.324100 24851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:20.324955 24866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:20.325145 24851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:20.325253 24851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "e955499aef4748a5962b56066e93bc71"
format_stamp: "Formatted at 2026-09-25 10:17:20 on dist-test-slave-dzwq"
server_key: "ab220b310c6f54c1039f6cfe9063ac3b"
server_key_iv: "e5945c0560d677887e1d2668deec788f"
server_key_version: "encryptionkey@0"
I20260925 10:17:20.325332 24851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:20.328850 24851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:20.331264 24851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:20.331429 24851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:20.335913 24851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36863
I20260925 10:17:20.335945 24918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36863 every 8 connection(s)
I20260925 10:17:20.336336 24851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:20.337347 24919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:20.339634 24919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: Bootstrap starting.
I20260925 10:17:20.340279 24919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:20.340518 24919 log.cc:824] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:20.341189 24919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: No bootstrap required, opened a new log
I20260925 10:17:20.342777 24919 raft_consensus.cc:348] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 10:17:20.342916 24919 raft_consensus.cc:374] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:20.342940 24919 raft_consensus.cc:729] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e955499aef4748a5962b56066e93bc71, State: Initialized, Role: FOLLOWER
I20260925 10:17:20.343047 24919 consensus_queue.cc:260] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 10:17:20.343101 24919 raft_consensus.cc:388] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:20.343122 24919 raft_consensus.cc:482] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:20.343209 24919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:20.343771 24919 raft_consensus.cc:504] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 10:17:20.343863 24919 leader_election.cc:302] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e955499aef4748a5962b56066e93bc71; no voters:
I20260925 10:17:20.344144 24919 leader_election.cc:288] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:20.344190 24922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:20.344277 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24851
I20260925 10:17:20.344415 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:20.344406 24922 raft_consensus.cc:686] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [term 1 LEADER]: Becoming Leader. State: Replica: e955499aef4748a5962b56066e93bc71, State: Running, Role: LEADER
I20260925 10:17:20.344475 24919 sys_catalog.cc:564] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:20.344499 24922 consensus_queue.cc:237] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } }
I20260925 10:17:20.344586 3683 external_mini_cluster.cc:1428] Setting key 8108211b26457eeb29b546d4ba498611
I20260925 10:17:20.344928 24924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e955499aef4748a5962b56066e93bc71. Latest consensus state: current_term: 1 leader_uuid: "e955499aef4748a5962b56066e93bc71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } } }
I20260925 10:17:20.345036 24924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:20.345304 24923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e955499aef4748a5962b56066e93bc71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e955499aef4748a5962b56066e93bc71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36863 } } }
I20260925 10:17:20.345479 24923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71 [sys.catalog]: This master's current role is: LEADER
W20260925 10:17:20.346782 24939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:20.346968 24939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:20.347216 24927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:20.347465 24927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:20.347470 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:20.345114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:59255 (local address 127.0.0.1:36863)
0925 10:17:20.345336 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:20.345347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:20.345407 (+ 60us) server_negotiation.cc:407] Connection header received
0925 10:17:20.345588 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:20.345596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:20.345640 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:20.345760 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:20.346923 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:20.346926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:20.346926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:20.346931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:20.346983 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:20.346998 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:20.347036 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:20.347176 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 10:17:20.347254 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260925 10:17:20.348565 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:20.350721 24927 catalog_manager.cc:1347] Generated new cluster ID: 397d0e31ac124fa3ad99eca511d71806
I20260925 10:17:20.350775 24927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:20.351332 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:20.350125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:59769 (local address 127.0.0.1:36863)
0925 10:17:20.350163 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:20.350165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:20.350301 (+ 136us) server_negotiation.cc:407] Connection header received
0925 10:17:20.350411 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:20.350412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:20.350417 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:20.350472 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:20.351112 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:20.351114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:20.351114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:20.351118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:20.351133 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:20.351137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:20.351186 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:20.351238 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 10:17:20.351252 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:20.371245 24927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:20.371479 24927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:20.377104 24927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e955499aef4748a5962b56066e93bc71: Generated new TSK 0
I20260925 10:17:20.377339 24927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:20.381026 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24851
2026-09-25T10:17:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:20 dist-test-slave-dzwq krb5kdc[24950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:20 dist-test-slave-dzwq krb5kdc[24950](info): set up 2 sockets
Sep 25 10:17:20 dist-test-slave-dzwq krb5kdc[24950](info): commencing operation
krb5kdc: starting...
W20260925 10:17:21.420817 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[24950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:21.547439 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:21.630999 24963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:21.631215 24963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:21.631238 24963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:21.632859 24963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:21.632903 24963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:21.632918 24963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:21.632957 24963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:21.632985 24963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:21.635247 24963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:21.635522 24963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:21.635777 24963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:21.640785 24968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:21.640772 24971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:21.640774 24969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:21.641008 24963 server_base.cc:1034] running on GCE node
I20260925 10:17:21.641290 24963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:21.641592 24963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:21.642758 24963 hybrid_clock.cc:648] HybridClock initialized: now 1790331441642748 us; error 50 us; skew 500 ppm
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[24950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:21.645262 24963 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:21.645704 24963 webserver.cc:460] Webserver started at http://127.0.0.1:41435/ using document root <none> and password file <none>
I20260925 10:17:21.645951 24963 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:21.646020 24963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:21.646169 24963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:21.647235 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "14fe05963e1d42acb6b1badda7f8aa14"
format_stamp: "Formatted at 2026-09-25 10:17:21 on dist-test-slave-dzwq"
server_key: "cb08934fc36646bd4d630a0991a0322c"
server_key_iv: "4a9ee642169f3ab8d2445e5afc340151"
server_key_version: "encryptionkey@0"
I20260925 10:17:21.647607 24963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "14fe05963e1d42acb6b1badda7f8aa14"
format_stamp: "Formatted at 2026-09-25 10:17:21 on dist-test-slave-dzwq"
server_key: "cb08934fc36646bd4d630a0991a0322c"
server_key_iv: "4a9ee642169f3ab8d2445e5afc340151"
server_key_version: "encryptionkey@0"
I20260925 10:17:21.649165 24963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:17:21.649994 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:21.650185 24963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:21.650285 24963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "14fe05963e1d42acb6b1badda7f8aa14"
format_stamp: "Formatted at 2026-09-25 10:17:21 on dist-test-slave-dzwq"
server_key: "cb08934fc36646bd4d630a0991a0322c"
server_key_iv: "4a9ee642169f3ab8d2445e5afc340151"
server_key_version: "encryptionkey@0"
I20260925 10:17:21.650367 24963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:21.662496 24963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:21.664402 24963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:21.664566 24963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:21.668507 24963 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35219
I20260925 10:17:21.668545 25030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35219 every 8 connection(s)
I20260925 10:17:21.668936 24963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:21.669950 25031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:21.672320 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 24963
I20260925 10:17:21.672380 25031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: Bootstrap starting.
I20260925 10:17:21.672451 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:21.672642 3683 external_mini_cluster.cc:1428] Setting key e122b965e94c6c9767492023bb8a1806
I20260925 10:17:21.673143 25031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:21.673475 25031 log.cc:824] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:21.674242 25031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: No bootstrap required, opened a new log
I20260925 10:17:21.676523 25031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260925 10:17:21.676702 25031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:21.676733 25031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14fe05963e1d42acb6b1badda7f8aa14, State: Initialized, Role: FOLLOWER
I20260925 10:17:21.676857 25031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260925 10:17:21.676951 25031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:21.676988 25031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:21.677038 25031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:21.677670 25031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260925 10:17:21.677784 25031 leader_election.cc:302] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14fe05963e1d42acb6b1badda7f8aa14; no voters:
I20260925 10:17:21.678306 25031 leader_election.cc:288] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:21.678368 25036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:21.678517 25036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [term 1 LEADER]: Becoming Leader. State: Replica: 14fe05963e1d42acb6b1badda7f8aa14, State: Running, Role: LEADER
I20260925 10:17:21.678589 25031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:21.678685 25036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260925 10:17:21.679013 25036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14fe05963e1d42acb6b1badda7f8aa14. Latest consensus state: current_term: 1 leader_uuid: "14fe05963e1d42acb6b1badda7f8aa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } } }
I20260925 10:17:21.679025 25038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14fe05963e1d42acb6b1badda7f8aa14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14fe05963e1d42acb6b1badda7f8aa14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } } }
I20260925 10:17:21.679107 25036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:21.679114 25038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14 [sys.catalog]: This master's current role is: LEADER
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[24950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260925 10:17:21.680663 25051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:21.680765 25051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:21.680891 25040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:21.681157 25040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:21.684377 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:21.673176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:38717 (local address 127.0.0.1:35219)
0925 10:17:21.673341 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:21.673349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:21.673374 (+ 25us) server_negotiation.cc:407] Connection header received
0925 10:17:21.673988 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:21.673996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:21.674050 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:21.674177 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:21.674925 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:21.675791 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:21.676953 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:21.677449 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:21.677538 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:21.680650 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:21.680654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:21.680654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:21.680666 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:21.682582 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:21.683073 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:21.683075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:21.683076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:21.683162 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:21.683414 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:21.683415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:21.683415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:21.683682 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:21.683800 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:21.684054 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 10:17:21.684159 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260925 10:17:21.685144 25040 catalog_manager.cc:1347] Generated new cluster ID: dd614b91d8d64413994d29e73ca6f618
I20260925 10:17:21.685290 25040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:21.685375 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:21.703712 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:21.687010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:59309 (local address 127.0.0.1:35219)
0925 10:17:21.687073 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:21.687075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:21.696180 (+ 9105us) server_negotiation.cc:407] Connection header received
0925 10:17:21.696271 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:21.696273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:21.696286 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:21.696342 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:21.697064 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:21.697878 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:21.698882 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:21.699241 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:21.699318 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:21.700776 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:21.700778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:21.700778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:21.700785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:21.702457 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:21.702875 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:21.702877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:21.702877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:21.702940 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:21.703174 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:21.703174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:21.703175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:21.703310 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:21.703442 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:21.703533 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 10:17:21.703606 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 10:17:21.705780 25040 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:21.706323 25040 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:21.710695 25040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14fe05963e1d42acb6b1badda7f8aa14: Generated new TSK 0
I20260925 10:17:21.710889 25040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:21.718538 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 24963
W20260925 10:17:21.723270 24948 connection.cc:535] client connection to 127.0.0.1:35219 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35219): Connection reset by peer (error 104)
2026-09-25T10:17:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[25062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[25062](info): set up 2 sockets
Sep 25 10:17:21 dist-test-slave-dzwq krb5kdc[25062](info): commencing operation
krb5kdc: starting...
W20260925 10:17:22.752847 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:22 dist-test-slave-dzwq krb5kdc[25062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:22.904771 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:22.987097 25075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:22.987321 25075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:22.987345 25075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:22.988468 25075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:22.988507 25075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:22.988520 25075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:22.988533 25075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:22.988545 25075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:22.990068 25075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36195
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:22.990274 25075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:22.990448 25075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:22.995443 25081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:22.995450 25080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:22.995450 25083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:22.995847 25075 server_base.cc:1034] running on GCE node
I20260925 10:17:22.996140 25075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:22.996448 25075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:22.997643 25075 hybrid_clock.cc:648] HybridClock initialized: now 1790331442997626 us; error 26 us; skew 500 ppm
Sep 25 10:17:22 dist-test-slave-dzwq krb5kdc[25062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:23.000229 25075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:23.000684 25075 webserver.cc:460] Webserver started at http://127.0.0.1:42007/ using document root <none> and password file <none>
I20260925 10:17:23.000913 25075 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:23.000983 25075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:23.001142 25075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:23.002303 25075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "98c45d4188484654b516f398aa1dc6c2"
format_stamp: "Formatted at 2026-09-25 10:17:22 on dist-test-slave-dzwq"
server_key: "615b8b7057f7fe4d6d4b2e76858975ce"
server_key_iv: "d664cdc52659a7f9a1dbf0e78f25b02f"
server_key_version: "encryptionkey@0"
I20260925 10:17:23.002682 25075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "98c45d4188484654b516f398aa1dc6c2"
format_stamp: "Formatted at 2026-09-25 10:17:22 on dist-test-slave-dzwq"
server_key: "615b8b7057f7fe4d6d4b2e76858975ce"
server_key_iv: "d664cdc52659a7f9a1dbf0e78f25b02f"
server_key_version: "encryptionkey@0"
I20260925 10:17:23.004055 25075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 10:17:23.004794 25090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:23.005014 25075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:23.005179 25075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "98c45d4188484654b516f398aa1dc6c2"
format_stamp: "Formatted at 2026-09-25 10:17:22 on dist-test-slave-dzwq"
server_key: "615b8b7057f7fe4d6d4b2e76858975ce"
server_key_iv: "d664cdc52659a7f9a1dbf0e78f25b02f"
server_key_version: "encryptionkey@0"
I20260925 10:17:23.005266 25075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:23.021473 25075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:23.023445 25075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:23.023623 25075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:23.027825 25075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36195
I20260925 10:17:23.027874 25142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36195 every 8 connection(s)
I20260925 10:17:23.028276 25075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:23.029232 25143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:23.029986 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25075
I20260925 10:17:23.030092 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:23.030323 3683 external_mini_cluster.cc:1428] Setting key 4b71a15a7dddd4674761045cafa35fe4
I20260925 10:17:23.032564 25143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2: Bootstrap starting.
I20260925 10:17:23.033236 25143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:23.033553 25143 log.cc:824] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:23.034312 25143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2: No bootstrap required, opened a new log
I20260925 10:17:23.036290 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:23.030950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:47017 (local address 127.0.0.1:36195)
0925 10:17:23.031144 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:23.031154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:23.031180 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:17:23.031680 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:23.031688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:23.031742 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:23.031865 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:23.032137 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:23.033160 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:23.034248 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:23.034841 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:23.034964 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:23.035678 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:23.035682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:23.035683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:23.035844 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:23.035875 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:23.035886 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:23.036002 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:23.036085 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 10:17:23.036123 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260925 10:17:23.036914 25143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 10:17:23.037127 25143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:23.037180 25143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c45d4188484654b516f398aa1dc6c2, State: Initialized, Role: FOLLOWER
I20260925 10:17:23.037323 25143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 10:17:23.037434 25143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:23.037479 25143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:23.037530 25143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:23.037599 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:23.038163 25143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 10:17:23.038286 25143 leader_election.cc:302] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c45d4188484654b516f398aa1dc6c2; no voters:
I20260925 10:17:23.038465 25143 leader_election.cc:288] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:23.038511 25149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:23.038674 25149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [term 1 LEADER]: Becoming Leader. State: Replica: 98c45d4188484654b516f398aa1dc6c2, State: Running, Role: LEADER
I20260925 10:17:23.038746 25143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:23.038807 25149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } }
I20260925 10:17:23.039318 25149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c45d4188484654b516f398aa1dc6c2. Latest consensus state: current_term: 1 leader_uuid: "98c45d4188484654b516f398aa1dc6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260925 10:17:23.039332 25154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c45d4188484654b516f398aa1dc6c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c45d4188484654b516f398aa1dc6c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36195 } } }
I20260925 10:17:23.039461 25149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:23.039533 25154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:23.040177 25162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:23.040918 25162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:23.042619 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:23.038917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:45653 (local address 127.0.0.1:36195)
0925 10:17:23.038956 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:23.038959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:23.039062 (+ 103us) server_negotiation.cc:407] Connection header received
0925 10:17:23.039168 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:23.039168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:23.039179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:23.039228 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:23.039380 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:23.040033 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:23.041142 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:23.041426 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:23.041491 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:23.042350 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:23.042351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:23.042352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:23.042356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:23.042371 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:23.042376 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:23.042431 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:23.042538 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 10:17:23.042551 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:17:23.042829 25162 catalog_manager.cc:1347] Generated new cluster ID: 768234e38bef47f28857d018f11c2c8a
I20260925 10:17:23.042877 25162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:23.068892 25162 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:23.069692 25162 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:23.075606 25162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c45d4188484654b516f398aa1dc6c2: Generated new TSK 0
I20260925 10:17:23.075819 25162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:23.085989 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25075
W20260925 10:17:23.090631 25060 connection.cc:535] client connection to 127.0.0.1:36195 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36195): Connection reset by peer (error 104)
2026-09-25T10:17:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:23 dist-test-slave-dzwq krb5kdc[25174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:23 dist-test-slave-dzwq krb5kdc[25174](info): set up 2 sockets
Sep 25 10:17:23 dist-test-slave-dzwq krb5kdc[25174](info): commencing operation
krb5kdc: starting...
W20260925 10:17:24.136850 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:24 dist-test-slave-dzwq krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 10:17:24.263659 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:24.347195 25187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:24.347431 25187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:24.347457 25187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:24.348644 25187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:24.348685 25187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:24.348699 25187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:24.348714 25187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:24.348726 25187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:24.350172 25187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:24.350381 25187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:24.350566 25187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:24.355600 25195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:24.355603 25192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:24.356006 25193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:24.356138 25187 server_base.cc:1034] running on GCE node
I20260925 10:17:24.356389 25187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:24.356689 25187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:24.357864 25187 hybrid_clock.cc:648] HybridClock initialized: now 1790331444357851 us; error 50 us; skew 500 ppm
Sep 25 10:17:24 dist-test-slave-dzwq krb5kdc[25174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:24.360455 25187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 10:17:24.360954 25187 webserver.cc:460] Webserver started at http://127.0.0.1:34081/ using document root <none> and password file <none>
I20260925 10:17:24.361202 25187 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:24.361274 25187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:24.361418 25187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:24.362509 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "8fb9b604826242e2bfa8903cc0f75c81"
format_stamp: "Formatted at 2026-09-25 10:17:24 on dist-test-slave-dzwq"
server_key: "97227827ccf6da8ae439dc80f79c671d"
server_key_iv: "797e3ac8514b88b78f543a49f560cab3"
server_key_version: "encryptionkey@0"
I20260925 10:17:24.362882 25187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "8fb9b604826242e2bfa8903cc0f75c81"
format_stamp: "Formatted at 2026-09-25 10:17:24 on dist-test-slave-dzwq"
server_key: "97227827ccf6da8ae439dc80f79c671d"
server_key_iv: "797e3ac8514b88b78f543a49f560cab3"
server_key_version: "encryptionkey@0"
I20260925 10:17:24.364317 25187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 10:17:24.365129 25202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:24.365336 25187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:24.365445 25187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "8fb9b604826242e2bfa8903cc0f75c81"
format_stamp: "Formatted at 2026-09-25 10:17:24 on dist-test-slave-dzwq"
server_key: "97227827ccf6da8ae439dc80f79c671d"
server_key_iv: "797e3ac8514b88b78f543a49f560cab3"
server_key_version: "encryptionkey@0"
I20260925 10:17:24.365525 25187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:24.368995 25187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:24.371383 25187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:24.371553 25187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:24.376145 25187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44615
I20260925 10:17:24.376170 25254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44615 every 8 connection(s)
I20260925 10:17:24.376525 25187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:24.377370 25255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:24.378680 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25187
I20260925 10:17:24.378800 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:24.379019 3683 external_mini_cluster.cc:1428] Setting key bd08520de6dcf0a0ce13f6aaddb64d37
I20260925 10:17:24.380339 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81: Bootstrap starting.
I20260925 10:17:24.381086 25255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:24.381474 25255 log.cc:824] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:24.381698 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:24.379565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:46997 (local address 127.0.0.1:44615)
0925 10:17:24.379764 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:24.379773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:24.379807 (+ 34us) server_negotiation.cc:407] Connection header received
0925 10:17:24.380294 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:24.380301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:24.380329 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:24.380499 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:24.381210 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:24.381214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:24.381214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:24.381219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:24.381264 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:24.381274 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:24.381341 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:24.381420 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 10:17:24.381477 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":80,"threads_started":1}
I20260925 10:17:24.382352 25255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81: No bootstrap required, opened a new log
I20260925 10:17:24.382895 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:24.384673 25255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260925 10:17:24.384820 25255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:24.384843 25255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb9b604826242e2bfa8903cc0f75c81, State: Initialized, Role: FOLLOWER
I20260925 10:17:24.384972 25255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260925 10:17:24.385056 25255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:24.385080 25255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:24.385159 25255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:24.385759 25255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260925 10:17:24.385881 25255 leader_election.cc:302] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fb9b604826242e2bfa8903cc0f75c81; no voters:
I20260925 10:17:24.386066 25255 leader_election.cc:288] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:24.386106 25265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:24.386325 25255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:24.386335 25265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [term 1 LEADER]: Becoming Leader. State: Replica: 8fb9b604826242e2bfa8903cc0f75c81, State: Running, Role: LEADER
I20260925 10:17:24.386739 25265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } }
I20260925 10:17:24.387372 25266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fb9b604826242e2bfa8903cc0f75c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260925 10:17:24.387506 25266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:24.387717 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:24.384376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.190:49575 (local address 127.0.0.1:44615)
0925 10:17:24.386443 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:24.386445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:24.386455 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:24.386476 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:24.386477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:24.386481 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:24.386546 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:24.387324 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:24.387326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:24.387326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:24.387331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:24.387354 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:24.387360 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:24.387598 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:24.387621 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 10:17:24.387636 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2046}
I20260925 10:17:24.387892 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb9b604826242e2bfa8903cc0f75c81. Latest consensus state: current_term: 1 leader_uuid: "8fb9b604826242e2bfa8903cc0f75c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb9b604826242e2bfa8903cc0f75c81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44615 } } }
I20260925 10:17:24.388039 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:24.388443 25275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:24.388893 25275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:24.390409 25275 catalog_manager.cc:1347] Generated new cluster ID: e1671e04ae7744ba9036bc06b0082678
I20260925 10:17:24.390465 25275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:24.407752 25275 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:24.408015 25275 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:24.423045 25275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fb9b604826242e2bfa8903cc0f75c81: Generated new TSK 0
I20260925 10:17:24.423311 25275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:24.427977 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25187
2026-09-25T10:17:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:24 dist-test-slave-dzwq krb5kdc[25286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:24 dist-test-slave-dzwq krb5kdc[25286](info): set up 2 sockets
Sep 25 10:17:24 dist-test-slave-dzwq krb5kdc[25286](info): commencing operation
krb5kdc: starting...
W20260925 10:17:25.500890 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:25.632861 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46481
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:25.717432 25299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:25.717648 25299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:25.717672 25299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:25.718773 25299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:25.718812 25299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:25.718827 25299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:25.718839 25299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:25.718851 25299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:25.720336 25299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:46481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:25.720535 25299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:25.720713 25299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:25.725788 25304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:25.725718 25307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:25.725772 25299 server_base.cc:1034] running on GCE node
W20260925 10:17:25.725718 25305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:25.726325 25299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:25.726649 25299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:25.727850 25299 hybrid_clock.cc:648] HybridClock initialized: now 1790331445727845 us; error 42 us; skew 500 ppm
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:25.739907 25299 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:25.740451 25299 webserver.cc:460] Webserver started at http://127.3.152.254:35479/ using document root <none> and password file <none>
I20260925 10:17:25.740700 25299 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:25.740774 25299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:25.740939 25299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:25.742118 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "276adc16afab46be89ed33ed911dfc60"
format_stamp: "Formatted at 2026-09-25 10:17:25 on dist-test-slave-dzwq"
server_key: "caf09324ba82b0d40adfb7e2a3735fcf"
server_key_iv: "0538c3514fddf6d9ea1af31c59ea3ca8"
server_key_version: "encryptionkey@0"
I20260925 10:17:25.742606 25299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "276adc16afab46be89ed33ed911dfc60"
format_stamp: "Formatted at 2026-09-25 10:17:25 on dist-test-slave-dzwq"
server_key: "caf09324ba82b0d40adfb7e2a3735fcf"
server_key_iv: "0538c3514fddf6d9ea1af31c59ea3ca8"
server_key_version: "encryptionkey@0"
I20260925 10:17:25.744330 25299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 10:17:25.745280 25314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:25.745466 25299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:25.745635 25299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "276adc16afab46be89ed33ed911dfc60"
format_stamp: "Formatted at 2026-09-25 10:17:25 on dist-test-slave-dzwq"
server_key: "caf09324ba82b0d40adfb7e2a3735fcf"
server_key_iv: "0538c3514fddf6d9ea1af31c59ea3ca8"
server_key_version: "encryptionkey@0"
I20260925 10:17:25.745724 25299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:25.753481 25299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:25.755653 25299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:25.755856 25299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:25.760444 25299 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:46481
I20260925 10:17:25.760490 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:46481 every 8 connection(s)
I20260925 10:17:25.760885 25299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:25.761757 25367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:25.764154 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: Bootstrap starting.
I20260925 10:17:25.764739 25367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:25.765035 25367 log.cc:824] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:25.765884 25367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: No bootstrap required, opened a new log
I20260925 10:17:25.767923 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25299
I20260925 10:17:25.767834 25367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } }
I20260925 10:17:25.768061 25367 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:25.768117 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:25.768119 25367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276adc16afab46be89ed33ed911dfc60, State: Initialized, Role: FOLLOWER
I20260925 10:17:25.768316 3683 external_mini_cluster.cc:1428] Setting key e0dab90e90a89afe20f59dc8895975e5
I20260925 10:17:25.768294 25367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } }
I20260925 10:17:25.768378 25367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:25.768424 25367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:25.768479 25367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:25.769250 25367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } }
I20260925 10:17:25.769387 25367 leader_election.cc:302] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276adc16afab46be89ed33ed911dfc60; no voters:
I20260925 10:17:25.769573 25367 leader_election.cc:288] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:25.769605 25372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:25.769804 25372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [term 1 LEADER]: Becoming Leader. State: Replica: 276adc16afab46be89ed33ed911dfc60, State: Running, Role: LEADER
I20260925 10:17:25.769860 25367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:25.769992 25372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } }
I20260925 10:17:25.770448 25373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276adc16afab46be89ed33ed911dfc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } } }
I20260925 10:17:25.770527 25373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:25.770963 25374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276adc16afab46be89ed33ed911dfc60. Latest consensus state: current_term: 1 leader_uuid: "276adc16afab46be89ed33ed911dfc60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276adc16afab46be89ed33ed911dfc60" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 46481 } } }
I20260925 10:17:25.771080 25374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60 [sys.catalog]: This master's current role is: LEADER
W20260925 10:17:25.772008 25388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:25.772089 25388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:25.772197 25380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:25.772372 25380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:25.774040 25380 catalog_manager.cc:1347] Generated new cluster ID: a92e743c260046139fa5c198011442e4
I20260925 10:17:25.774106 25380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:25.782166 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:25.768841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44657 (local address 127.3.152.254:46481)
0925 10:17:25.769338 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:25.769353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:25.769400 (+ 47us) server_negotiation.cc:407] Connection header received
0925 10:17:25.769538 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:25.769544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:25.769602 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:25.769753 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:25.770749 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:25.772289 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:25.773410 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:25.774176 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:25.774300 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:25.777330 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:25.777335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:25.777335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:25.777350 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:25.780169 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:25.780648 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:25.780652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:25.780652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:25.780778 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:25.781050 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:25.781051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:25.781051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:25.781407 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:25.781574 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:25.781701 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 10:17:25.781930 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":418,"threads_started":1}
I20260925 10:17:25.783226 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:25.785213 25380 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:25.785745 25380 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:25.792490 25371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:25.784774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56435 (local address 127.3.152.254:46481)
0925 10:17:25.784809 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:25.784811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:25.785077 (+ 266us) server_negotiation.cc:407] Connection header received
0925 10:17:25.785142 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:25.785143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:25.785152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:25.785207 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:25.785899 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:25.786687 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:25.787724 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:25.788097 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:25.788165 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:25.789559 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:25.789560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:25.789560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:25.789567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:25.791202 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:25.791619 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:25.791620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:25.791621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:25.791678 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:25.791912 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:25.791913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:25.791913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:25.792070 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:25.792172 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:25.792333 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 10:17:25.792397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:17:25.796900 25380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276adc16afab46be89ed33ed911dfc60: Generated new TSK 0
I20260925 10:17:25.797101 25380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:25.801867 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25299
W20260925 10:17:25.806248 25284 connection.cc:535] client connection to 127.3.152.254:46481 recv error: Network error: failed to read from TLS socket (remote: 127.3.152.254:46481): Connection reset by peer (error 104)
2026-09-25T10:17:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25398](info): set up 2 sockets
Sep 25 10:17:25 dist-test-slave-dzwq krb5kdc[25398](info): commencing operation
krb5kdc: starting...
W20260925 10:17:26.884878 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:26 dist-test-slave-dzwq krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:27.011765 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43919
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:27.093941 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:27.094225 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:27.094270 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:27.095448 25411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:27.095538 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:27.095564 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:27.095593 25411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:27.095635 25411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:27.097235 25411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:43919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:27.097535 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:27.097779 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:27.102653 25417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:27.102855 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:27.103003 25411 server_base.cc:1034] running on GCE node
W20260925 10:17:27.102854 25416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:27.103384 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:27.103653 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:27.104810 25411 hybrid_clock.cc:648] HybridClock initialized: now 1790331447104743 us; error 79 us; skew 500 ppm
Sep 25 10:17:27 dist-test-slave-dzwq krb5kdc[25398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:27.107405 25411 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:27.107853 25411 webserver.cc:460] Webserver started at http://127.3.152.254:36849/ using document root <none> and password file <none>
I20260925 10:17:27.108148 25411 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:27.108223 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:27.108381 25411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:27.109505 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "c325835e18954e84a57841072bef0311"
format_stamp: "Formatted at 2026-09-25 10:17:27 on dist-test-slave-dzwq"
server_key: "5ca4e101fddf712e32fd9f73d8ededf2"
server_key_iv: "56deb7f420b2dbb08f1848d9735508ee"
server_key_version: "encryptionkey@0"
I20260925 10:17:27.109891 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "c325835e18954e84a57841072bef0311"
format_stamp: "Formatted at 2026-09-25 10:17:27 on dist-test-slave-dzwq"
server_key: "5ca4e101fddf712e32fd9f73d8ededf2"
server_key_iv: "56deb7f420b2dbb08f1848d9735508ee"
server_key_version: "encryptionkey@0"
I20260925 10:17:27.111263 25411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:27.112171 25426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:27.112370 25411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:27.112473 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "c325835e18954e84a57841072bef0311"
format_stamp: "Formatted at 2026-09-25 10:17:27 on dist-test-slave-dzwq"
server_key: "5ca4e101fddf712e32fd9f73d8ededf2"
server_key_iv: "56deb7f420b2dbb08f1848d9735508ee"
server_key_version: "encryptionkey@0"
I20260925 10:17:27.112567 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:27.133911 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:27.136433 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:27.136613 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:27.140599 25411 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:43919
I20260925 10:17:27.140653 25478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:43919 every 8 connection(s)
I20260925 10:17:27.141048 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:27.141914 25479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:27.144260 25479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311: Bootstrap starting.
I20260925 10:17:27.144874 25479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:27.145179 25479 log.cc:824] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:27.145913 25479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311: No bootstrap required, opened a new log
I20260925 10:17:27.147289 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25411
I20260925 10:17:27.147387 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:27.147627 3683 external_mini_cluster.cc:1428] Setting key 768ecb2bd7f55b0418d7b559f2c7c7d8
I20260925 10:17:27.147651 25479 raft_consensus.cc:348] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } }
I20260925 10:17:27.147854 25479 raft_consensus.cc:374] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:27.147902 25479 raft_consensus.cc:729] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325835e18954e84a57841072bef0311, State: Initialized, Role: FOLLOWER
I20260925 10:17:27.148073 25479 consensus_queue.cc:260] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } }
I20260925 10:17:27.148175 25479 raft_consensus.cc:388] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:27.148219 25479 raft_consensus.cc:482] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:27.148314 25479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:27.149341 25479 raft_consensus.cc:504] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } }
I20260925 10:17:27.149442 25479 leader_election.cc:302] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c325835e18954e84a57841072bef0311; no voters:
I20260925 10:17:27.149668 25479 leader_election.cc:288] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:27.149700 25484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:27.149806 25484 raft_consensus.cc:686] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [term 1 LEADER]: Becoming Leader. State: Replica: c325835e18954e84a57841072bef0311, State: Running, Role: LEADER
I20260925 10:17:27.149940 25479 sys_catalog.cc:564] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:27.149950 25484 consensus_queue.cc:237] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } }
I20260925 10:17:27.150404 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c325835e18954e84a57841072bef0311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } } }
I20260925 10:17:27.150437 25486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c325835e18954e84a57841072bef0311. Latest consensus state: current_term: 1 leader_uuid: "c325835e18954e84a57841072bef0311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325835e18954e84a57841072bef0311" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 43919 } } }
I20260925 10:17:27.150637 25486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:27.150854 25493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:27.150542 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:27.151445 25493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:27.153131 25493 catalog_manager.cc:1347] Generated new cluster ID: c2441feefc804b0c8da3d450bc269cba
I20260925 10:17:27.153216 25493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:27.153891 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:27.148415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46849 (local address 127.3.152.254:43919)
0925 10:17:27.148617 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:27.148627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:27.148662 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:17:27.148776 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:27.148782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:27.148824 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:27.148941 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:27.149239 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:27.150515 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:27.151709 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:27.152278 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:27.152421 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:27.153204 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:27.153207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:27.153208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:27.153412 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:27.153450 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:27.153465 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:27.153554 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:27.153661 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 10:17:27.153690 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260925 10:17:27.154851 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:27.159883 25483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:27.156402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42273 (local address 127.3.152.254:43919)
0925 10:17:27.156441 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:27.156443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:27.156498 (+ 55us) server_negotiation.cc:407] Connection header received
0925 10:17:27.156555 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:27.156556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:27.156565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:27.156632 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:27.156764 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:27.157518 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:27.158497 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:27.158839 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:27.158912 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:27.159627 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:27.159628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:27.159628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:27.159633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:27.159647 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:27.159652 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:27.159716 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:27.159798 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 10:17:27.159811 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:17:27.161054 25493 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:27.161587 25493 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:27.180408 25493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c325835e18954e84a57841072bef0311: Generated new TSK 0
I20260925 10:17:27.180629 25493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:27.185258 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25411
W20260925 10:17:27.189711 25396 connection.cc:535] client connection to 127.3.152.254:43919 recv error: Network error: failed to read from TLS socket (remote: 127.3.152.254:43919): Connection reset by peer (error 104)
2026-09-25T10:17:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1383 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:27 dist-test-slave-dzwq krb5kdc[25510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:27 dist-test-slave-dzwq krb5kdc[25510](info): set up 2 sockets
Sep 25 10:17:27 dist-test-slave-dzwq krb5kdc[25510](info): commencing operation
krb5kdc: starting...
W20260925 10:17:28.220808 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:28 dist-test-slave-dzwq krb5kdc[25510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:28.343130 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36985
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:28.425973 25523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:28.426285 25523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:28.426324 25523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:28.427541 25523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:28.427647 25523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:28.427672 25523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:28.427692 25523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:28.427726 25523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:28.429351 25523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:36985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:28.429672 25523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:28.429920 25523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:28.434933 25531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:28.434933 25528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:28.435005 25529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:28.435493 25523 server_base.cc:1034] running on GCE node
I20260925 10:17:28.435757 25523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:28.436120 25523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:28.437299 25523 hybrid_clock.cc:648] HybridClock initialized: now 1790331448437292 us; error 37 us; skew 500 ppm
Sep 25 10:17:28 dist-test-slave-dzwq krb5kdc[25510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331448, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:28.439901 25523 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:28.440392 25523 webserver.cc:460] Webserver started at http://127.3.152.254:41481/ using document root <none> and password file <none>
I20260925 10:17:28.440605 25523 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:28.440649 25523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:28.440742 25523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:28.441799 25523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "1ef3147537e34c598c6eac5aea2d0b0e"
format_stamp: "Formatted at 2026-09-25 10:17:28 on dist-test-slave-dzwq"
server_key: "af58e37d528424983926a4aec20d970d"
server_key_iv: "2dbf1a40b03ef9032aa00e77daeceb2f"
server_key_version: "encryptionkey@0"
I20260925 10:17:28.442114 25523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "1ef3147537e34c598c6eac5aea2d0b0e"
format_stamp: "Formatted at 2026-09-25 10:17:28 on dist-test-slave-dzwq"
server_key: "af58e37d528424983926a4aec20d970d"
server_key_iv: "2dbf1a40b03ef9032aa00e77daeceb2f"
server_key_version: "encryptionkey@0"
I20260925 10:17:28.443475 25523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:28.444289 25538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:28.444495 25523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:28.444589 25523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "1ef3147537e34c598c6eac5aea2d0b0e"
format_stamp: "Formatted at 2026-09-25 10:17:28 on dist-test-slave-dzwq"
server_key: "af58e37d528424983926a4aec20d970d"
server_key_iv: "2dbf1a40b03ef9032aa00e77daeceb2f"
server_key_version: "encryptionkey@0"
I20260925 10:17:28.444670 25523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:28.448136 25523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:28.450237 25523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:28.450349 25523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:28.454298 25523 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:36985
I20260925 10:17:28.454332 25590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:36985 every 8 connection(s)
I20260925 10:17:28.454738 25523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:28.455854 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:28.458173 25591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e: Bootstrap starting.
I20260925 10:17:28.458376 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25523
I20260925 10:17:28.458500 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:28.458722 3683 external_mini_cluster.cc:1428] Setting key 8572c95778ae0eb2130c8e84e827bd27
I20260925 10:17:28.458937 25591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:28.459299 25591 log.cc:824] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:28.460047 25591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e: No bootstrap required, opened a new log
I20260925 10:17:28.461373 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:28.459277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50017 (local address 127.3.152.254:36985)
0925 10:17:28.459504 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:28.459514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:28.459542 (+ 28us) server_negotiation.cc:407] Connection header received
0925 10:17:28.460048 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:28.460056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:28.460092 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:28.460253 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:28.460959 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:28.460962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:28.460962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:28.460967 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:28.461013 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:28.461023 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:28.461115 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:28.461152 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 10:17:28.461207 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260925 10:17:28.462476 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:28.462455 25591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } }
I20260925 10:17:28.462628 25591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:28.462674 25591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef3147537e34c598c6eac5aea2d0b0e, State: Initialized, Role: FOLLOWER
I20260925 10:17:28.462800 25591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } }
I20260925 10:17:28.462905 25591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:28.462950 25591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:28.463011 25591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:28.463606 25591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } }
I20260925 10:17:28.463721 25591 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef3147537e34c598c6eac5aea2d0b0e; no voters:
I20260925 10:17:28.463886 25591 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:28.464082 25602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:28.464164 25591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:28.464250 25602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [term 1 LEADER]: Becoming Leader. State: Replica: 1ef3147537e34c598c6eac5aea2d0b0e, State: Running, Role: LEADER
I20260925 10:17:28.464340 25602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } }
I20260925 10:17:28.464704 25602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef3147537e34c598c6eac5aea2d0b0e. Latest consensus state: current_term: 1 leader_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } } }
I20260925 10:17:28.464826 25602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:28.464713 25603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef3147537e34c598c6eac5aea2d0b0e" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 36985 } } }
I20260925 10:17:28.465016 25603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:28.465019 25610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:28.465562 25610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:28.466877 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:28.463930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60245 (local address 127.3.152.254:36985)
0925 10:17:28.463989 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:28.463990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:28.465712 (+ 1722us) server_negotiation.cc:407] Connection header received
0925 10:17:28.465894 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:28.465895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:28.465902 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:28.465932 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:28.466631 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:28.466632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:28.466633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:28.466637 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:28.466653 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:28.466658 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:28.466726 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:28.466799 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 10:17:28.466814 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:17:28.467188 25610 catalog_manager.cc:1347] Generated new cluster ID: 045349c1b75241898581ae1087045bc5
I20260925 10:17:28.467267 25610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:28.487695 25610 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:28.487902 25610 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:28.495041 25610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef3147537e34c598c6eac5aea2d0b0e: Generated new TSK 0
I20260925 10:17:28.495270 25610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:28.498790 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25523
2026-09-25T10:17:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:28 dist-test-slave-dzwq krb5kdc[25622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:28 dist-test-slave-dzwq krb5kdc[25622](info): set up 2 sockets
Sep 25 10:17:28 dist-test-slave-dzwq krb5kdc[25622](info): commencing operation
krb5kdc: starting...
W20260925 10:17:29.592760 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:29.717036 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33035
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:39451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:29.797983 25635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:29.798206 25635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:29.798230 25635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:29.799440 25635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:29.799485 25635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:29.799499 25635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:29.799511 25635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:29.799523 25635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:29.801101 25635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:39451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:29.801376 25635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:29.801656 25635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:29.806522 25641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:29.806548 25643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:29.806653 25640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:29.806875 25635 server_base.cc:1034] running on GCE node
I20260925 10:17:29.807140 25635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:29.807439 25635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:29.808585 25635 hybrid_clock.cc:648] HybridClock initialized: now 1790331449808564 us; error 40 us; skew 500 ppm
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331449, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:29.811612 25635 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:29.812166 25635 webserver.cc:460] Webserver started at http://127.3.152.254:45157/ using document root <none> and password file <none>
I20260925 10:17:29.812423 25635 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:29.812470 25635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:29.812641 25635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:29.813720 25635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "032faa742bd64fb29c4b9180367fdd09"
format_stamp: "Formatted at 2026-09-25 10:17:29 on dist-test-slave-dzwq"
server_key: "5832f798626e44e52690b9d11914e3f1"
server_key_iv: "6f16d76ef70c8031d4a93c867069db82"
server_key_version: "encryptionkey@0"
I20260925 10:17:29.814090 25635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "032faa742bd64fb29c4b9180367fdd09"
format_stamp: "Formatted at 2026-09-25 10:17:29 on dist-test-slave-dzwq"
server_key: "5832f798626e44e52690b9d11914e3f1"
server_key_iv: "6f16d76ef70c8031d4a93c867069db82"
server_key_version: "encryptionkey@0"
I20260925 10:17:29.815455 25635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:29.816295 25650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:29.816481 25635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:29.816577 25635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "032faa742bd64fb29c4b9180367fdd09"
format_stamp: "Formatted at 2026-09-25 10:17:29 on dist-test-slave-dzwq"
server_key: "5832f798626e44e52690b9d11914e3f1"
server_key_iv: "6f16d76ef70c8031d4a93c867069db82"
server_key_version: "encryptionkey@0"
I20260925 10:17:29.816690 25635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:29.844502 25635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:29.846750 25635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:29.846940 25635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:29.851238 25635 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:33035
I20260925 10:17:29.851413 25702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:33035 every 8 connection(s)
I20260925 10:17:29.851678 25635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:29.852367 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25635
I20260925 10:17:29.852490 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:29.852656 3683 external_mini_cluster.cc:1428] Setting key 7218ddb248446ecf0cba93fb333ec9db
I20260925 10:17:29.852896 25703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:29.856671 25703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09: Bootstrap starting.
I20260925 10:17:29.857345 25703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:29.857734 25703 log.cc:824] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:29.858587 25703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09: No bootstrap required, opened a new log
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:29.860988 25703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } }
I20260925 10:17:29.861184 25703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:29.861231 25703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032faa742bd64fb29c4b9180367fdd09, State: Initialized, Role: FOLLOWER
I20260925 10:17:29.861403 25703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } }
I20260925 10:17:29.861503 25703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:29.861541 25703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:29.861598 25703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:29.862426 25703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } }
I20260925 10:17:29.862555 25703 leader_election.cc:302] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032faa742bd64fb29c4b9180367fdd09; no voters:
I20260925 10:17:29.862727 25703 leader_election.cc:288] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:29.862797 25708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:29.862977 25708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [term 1 LEADER]: Becoming Leader. State: Replica: 032faa742bd64fb29c4b9180367fdd09, State: Running, Role: LEADER
I20260925 10:17:29.863046 25703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:29.863111 25708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } }
I20260925 10:17:29.863461 25708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032faa742bd64fb29c4b9180367fdd09. Latest consensus state: current_term: 1 leader_uuid: "032faa742bd64fb29c4b9180367fdd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } } }
I20260925 10:17:29.863590 25708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:29.863898 25716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:29.864477 25716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:29.865043 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:29.853254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55461 (local address 127.3.152.254:33035)
0925 10:17:29.853511 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:29.853521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:29.853558 (+ 37us) server_negotiation.cc:407] Connection header received
0925 10:17:29.854219 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:29.854229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:29.854299 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:29.854485 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:29.855151 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:29.856579 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:29.857427 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:29.858074 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:29.858166 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:29.860931 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:29.860935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:29.860936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:29.860949 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:29.863204 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:29.863684 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:29.863688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:29.863688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:29.863802 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:29.864049 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:29.864050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:29.864050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:29.864378 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:29.864534 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:29.864704 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 10:17:29.864843 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260925 10:17:29.865485 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032faa742bd64fb29c4b9180367fdd09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032faa742bd64fb29c4b9180367fdd09" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 33035 } } }
I20260925 10:17:29.865638 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:29.866080 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:29.866169 25716 catalog_manager.cc:1347] Generated new cluster ID: 32445f1052ed4b9bbb5b854a2d07467f
I20260925 10:17:29.866217 25716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:29.873682 25716 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:29.874398 25716 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:29.875561 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:29.867595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50959 (local address 127.3.152.254:33035)
0925 10:17:29.867632 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:29.867634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:29.867749 (+ 115us) server_negotiation.cc:407] Connection header received
0925 10:17:29.867796 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:29.867797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:29.867807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:29.867868 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:29.868551 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:29.869334 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:29.870356 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:29.870709 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:29.870784 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:29.872382 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:29.872383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:29.872384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:29.872391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:29.874226 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:29.874688 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:29.874689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:29.874690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:29.874757 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:29.874997 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:29.874998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:29.874999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:29.875116 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:29.875226 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:29.875351 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:17:29.875443 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 10:17:29.887125 25716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032faa742bd64fb29c4b9180367fdd09: Generated new TSK 0
I20260925 10:17:29.887383 25716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:29.891388 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25635
2026-09-25T10:17:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25734](info): set up 2 sockets
Sep 25 10:17:29 dist-test-slave-dzwq krb5kdc[25734](info): commencing operation
krb5kdc: starting...
W20260925 10:17:30.920837 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:31 dist-test-slave-dzwq krb5kdc[25734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:31.043015 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41819
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:31.125563 25747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:31.125782 25747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:31.125808 25747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:31.126912 25747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:31.126951 25747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:31.126965 25747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:31.126978 25747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:31.126993 25747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:31.128459 25747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:31.128667 25747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:31.128842 25747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:31.133869 25753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:31.133893 25752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:31.133982 25747 server_base.cc:1034] running on GCE node
W20260925 10:17:31.133870 25755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:31.134353 25747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:31.134661 25747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:31.135831 25747 hybrid_clock.cc:648] HybridClock initialized: now 1790331451135823 us; error 50 us; skew 500 ppm
Sep 25 10:17:31 dist-test-slave-dzwq krb5kdc[25734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331451, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:31.144188 25747 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:31.144655 25747 webserver.cc:460] Webserver started at http://127.3.152.254:39485/ using document root <none> and password file <none>
I20260925 10:17:31.144925 25747 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:31.144974 25747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:31.145145 25747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:31.146277 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "ac9c79cd5f304c7ca65d675e428abe42"
format_stamp: "Formatted at 2026-09-25 10:17:31 on dist-test-slave-dzwq"
server_key: "b15573d6a3e04c9c64c3e6a2f87336c1"
server_key_iv: "75005daff2613b70cc0fef83aadea0ae"
server_key_version: "encryptionkey@0"
I20260925 10:17:31.146668 25747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "ac9c79cd5f304c7ca65d675e428abe42"
format_stamp: "Formatted at 2026-09-25 10:17:31 on dist-test-slave-dzwq"
server_key: "b15573d6a3e04c9c64c3e6a2f87336c1"
server_key_iv: "75005daff2613b70cc0fef83aadea0ae"
server_key_version: "encryptionkey@0"
I20260925 10:17:31.148222 25747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 10:17:31.149010 25762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:31.149227 25747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:31.149329 25747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "ac9c79cd5f304c7ca65d675e428abe42"
format_stamp: "Formatted at 2026-09-25 10:17:31 on dist-test-slave-dzwq"
server_key: "b15573d6a3e04c9c64c3e6a2f87336c1"
server_key_iv: "75005daff2613b70cc0fef83aadea0ae"
server_key_version: "encryptionkey@0"
I20260925 10:17:31.149400 25747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:31.158715 25747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:31.161100 25747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:31.161262 25747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:31.165454 25747 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:41819
I20260925 10:17:31.165485 25814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:41819 every 8 connection(s)
I20260925 10:17:31.165864 25747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:31.166846 25815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:31.168063 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25747
I20260925 10:17:31.168192 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:31.168385 3683 external_mini_cluster.cc:1428] Setting key 9b7f59fc89ca66b64ee9cc88d2591ceb
I20260925 10:17:31.169837 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42: Bootstrap starting.
I20260925 10:17:31.170506 25815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:31.170814 25815 log.cc:824] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:31.171643 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42: No bootstrap required, opened a new log
I20260925 10:17:31.174297 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:31.168985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51893 (local address 127.3.152.254:41819)
0925 10:17:31.169166 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:31.169174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:31.169200 (+ 26us) server_negotiation.cc:407] Connection header received
0925 10:17:31.169633 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:31.169639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:31.169688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:31.169809 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:31.170077 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:31.171125 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:31.172079 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:31.172696 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:31.172811 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:31.173536 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:31.173541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:31.173542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:31.173785 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:31.173831 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:31.173848 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:31.173931 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:31.174016 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 10:17:31.174081 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260925 10:17:31.174268 25815 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } }
I20260925 10:17:31.174672 25815 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:31.174714 25815 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9c79cd5f304c7ca65d675e428abe42, State: Initialized, Role: FOLLOWER
I20260925 10:17:31.174844 25815 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } }
I20260925 10:17:31.174954 25815 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:31.174994 25815 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:31.175041 25815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:31.175210 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:31.175685 25815 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } }
I20260925 10:17:31.175774 25815 leader_election.cc:302] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac9c79cd5f304c7ca65d675e428abe42; no voters:
I20260925 10:17:31.175957 25815 leader_election.cc:288] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:31.176029 25820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:31.176192 25820 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [term 1 LEADER]: Becoming Leader. State: Replica: ac9c79cd5f304c7ca65d675e428abe42, State: Running, Role: LEADER
I20260925 10:17:31.176250 25815 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:31.176368 25820 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } }
I20260925 10:17:31.176828 25822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac9c79cd5f304c7ca65d675e428abe42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } } }
I20260925 10:17:31.176914 25822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:31.177006 25821 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac9c79cd5f304c7ca65d675e428abe42. Latest consensus state: current_term: 1 leader_uuid: "ac9c79cd5f304c7ca65d675e428abe42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9c79cd5f304c7ca65d675e428abe42" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 41819 } } }
I20260925 10:17:31.177275 25833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:31.177281 25821 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:31.177819 25833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:31.179638 25833 catalog_manager.cc:1347] Generated new cluster ID: 70376b72cf094ac0a2373d0eef2b3588
I20260925 10:17:31.179692 25833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:31.181439 25818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:31.177884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43029 (local address 127.3.152.254:41819)
0925 10:17:31.177923 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:31.177925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:31.178095 (+ 170us) server_negotiation.cc:407] Connection header received
0925 10:17:31.178243 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:31.178244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:31.178254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:31.178319 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:31.178534 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:31.179242 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:31.180171 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:31.180389 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:31.180478 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:31.181195 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:31.181196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:31.181197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:31.181201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:31.181216 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:31.181221 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:31.181275 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:31.181338 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 10:17:31.181361 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 10:17:31.188968 25833 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:31.189594 25833 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:31.196620 25833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac9c79cd5f304c7ca65d675e428abe42: Generated new TSK 0
I20260925 10:17:31.196830 25833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:31.199568 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25747
2026-09-25T10:17:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:31 dist-test-slave-dzwq krb5kdc[25846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:31 dist-test-slave-dzwq krb5kdc[25846](info): set up 2 sockets
Sep 25 10:17:31 dist-test-slave-dzwq krb5kdc[25846](info): commencing operation
krb5kdc: starting...
W20260925 10:17:32.244896 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:32 dist-test-slave-dzwq krb5kdc[25846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:32.367162 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37527
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:37527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:32.448863 25859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:32.449097 25859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:32.449121 25859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:32.450250 25859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:32.450287 25859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:32.450301 25859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:32.450314 25859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:32.450326 25859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:32.451738 25859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:37527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:32.451933 25859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:32.452234 25859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:32.457201 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:32.457250 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:32.457402 25859 server_base.cc:1034] running on GCE node
W20260925 10:17:32.457250 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:32.457732 25859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:32.458034 25859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:32.459235 25859 hybrid_clock.cc:648] HybridClock initialized: now 1790331452459210 us; error 34 us; skew 500 ppm
Sep 25 10:17:32 dist-test-slave-dzwq krb5kdc[25846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:32.462106 25859 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:32.462617 25859 webserver.cc:460] Webserver started at http://127.3.152.254:43287/ using document root <none> and password file <none>
I20260925 10:17:32.462880 25859 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:32.462955 25859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:32.463109 25859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:32.464282 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "b75e5705e4f1482caef18c8192857f4b"
format_stamp: "Formatted at 2026-09-25 10:17:32 on dist-test-slave-dzwq"
server_key: "316d6dec769c52ce7f36941e5e04e029"
server_key_iv: "f737bc1f26ef2361cd72679da89613fd"
server_key_version: "encryptionkey@0"
I20260925 10:17:32.464663 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "b75e5705e4f1482caef18c8192857f4b"
format_stamp: "Formatted at 2026-09-25 10:17:32 on dist-test-slave-dzwq"
server_key: "316d6dec769c52ce7f36941e5e04e029"
server_key_iv: "f737bc1f26ef2361cd72679da89613fd"
server_key_version: "encryptionkey@0"
I20260925 10:17:32.465994 25859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:32.466725 25874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:32.466912 25859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:32.467018 25859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "b75e5705e4f1482caef18c8192857f4b"
format_stamp: "Formatted at 2026-09-25 10:17:32 on dist-test-slave-dzwq"
server_key: "316d6dec769c52ce7f36941e5e04e029"
server_key_iv: "f737bc1f26ef2361cd72679da89613fd"
server_key_version: "encryptionkey@0"
I20260925 10:17:32.467093 25859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:32.470534 25859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:32.472992 25859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:32.473158 25859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:32.477267 25859 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:37527
I20260925 10:17:32.477300 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:37527 every 8 connection(s)
I20260925 10:17:32.477661 25859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:32.478682 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:32.481132 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b: Bootstrap starting.
I20260925 10:17:32.481779 25927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:32.481976 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25859
I20260925 10:17:32.482100 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:32.482121 25927 log.cc:824] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:32.482300 3683 external_mini_cluster.cc:1428] Setting key 1b4747c65cb678e4551cbe34742eca03
I20260925 10:17:32.483031 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b: No bootstrap required, opened a new log
I20260925 10:17:32.484874 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:32.482862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52157 (local address 127.3.152.254:37527)
0925 10:17:32.483029 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:32.483038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:32.483097 (+ 59us) server_negotiation.cc:407] Connection header received
0925 10:17:32.483543 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:32.483549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:32.483575 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:32.483696 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:32.484397 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:32.484400 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:32.484400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:32.484404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:32.484443 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:32.484453 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:32.484523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:32.484597 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 10:17:32.484653 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 10:17:32.485965 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:32.485868 25927 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } }
I20260925 10:17:32.486047 25927 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 10:17:32.486088 25927 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e5705e4f1482caef18c8192857f4b, State: Initialized, Role: FOLLOWER
I20260925 10:17:32.486212 25927 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } }
I20260925 10:17:32.486297 25927 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 10:17:32.486325 25927 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 10:17:32.486371 25927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:32.487155 25927 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } }
I20260925 10:17:32.487280 25927 leader_election.cc:302] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75e5705e4f1482caef18c8192857f4b; no voters:
I20260925 10:17:32.487438 25927 leader_election.cc:288] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 10:17:32.487460 25936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:32.487572 25936 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [term 1 LEADER]: Becoming Leader. State: Replica: b75e5705e4f1482caef18c8192857f4b, State: Running, Role: LEADER
I20260925 10:17:32.487699 25927 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:32.487697 25936 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } }
I20260925 10:17:32.488291 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75e5705e4f1482caef18c8192857f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } } }
I20260925 10:17:32.488310 25938 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75e5705e4f1482caef18c8192857f4b. Latest consensus state: current_term: 1 leader_uuid: "b75e5705e4f1482caef18c8192857f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75e5705e4f1482caef18c8192857f4b" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37527 } } }
I20260925 10:17:32.488426 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:32.488438 25938 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:32.488659 25946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:32.489003 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:32.487566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60441 (local address 127.3.152.254:37527)
0925 10:17:32.487604 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:32.487606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:32.487818 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:17:32.487919 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:32.487920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:32.487925 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:32.487954 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:32.488757 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:32.488758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:32.488759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 10:17:32.488763 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:32.488780 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 10:17:32.488785 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:32.488815 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:32.488923 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 10:17:32.488937 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 10:17:32.489504 25946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:32.491065 25946 catalog_manager.cc:1347] Generated new cluster ID: 7a27865e314b493fb8cec51ffd2467a9
I20260925 10:17:32.491107 25946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:32.507448 25946 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:32.507669 25946 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:32.525465 25946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75e5705e4f1482caef18c8192857f4b: Generated new TSK 0
I20260925 10:17:32.525722 25946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:32.534919 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25859
2026-09-25T10:17:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1335 ms)
[----------] 18 tests from AuthTokenIssuingTest (24230 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:32 dist-test-slave-dzwq krb5kdc[25958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:32 dist-test-slave-dzwq krb5kdc[25958](info): set up 2 sockets
Sep 25 10:17:32 dist-test-slave-dzwq krb5kdc[25958](info): commencing operation
krb5kdc: starting...
W20260925 10:17:33.568774 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:33.687770 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37235
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:33.770110 25971 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:33.770375 25971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:33.770406 25971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:33.770443 25971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:33.771601 25971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:33.771675 25971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:33.771710 25971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:33.771739 25971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:33.771760 25971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:33.773414 25971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:37235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:33.773689 25971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:33.773937 25971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:33.779069 25977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:33.779107 25979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:33.779127 25976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:33.779639 25971 server_base.cc:1034] running on GCE node
I20260925 10:17:33.779933 25971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:33.780270 25971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:33.781440 25971 hybrid_clock.cc:648] HybridClock initialized: now 1790331453781419 us; error 36 us; skew 500 ppm
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:33.784031 25971 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:33.784478 25971 webserver.cc:460] Webserver started at http://127.3.152.254:38115/ using document root <none> and password file <none>
I20260925 10:17:33.784667 25971 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:33.784713 25971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:33.784811 25971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:33.785846 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "aad22396b6e94086a82e060a3fc0c446"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "b7169c629e36e23d760a2bf51a2cae9a"
server_key_iv: "60c2791928ebabf58129bce1beda0a83"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.786160 25971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "aad22396b6e94086a82e060a3fc0c446"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "b7169c629e36e23d760a2bf51a2cae9a"
server_key_iv: "60c2791928ebabf58129bce1beda0a83"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.787441 25971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:33.788208 25986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:33.788409 25971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:33.788501 25971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "aad22396b6e94086a82e060a3fc0c446"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "b7169c629e36e23d760a2bf51a2cae9a"
server_key_iv: "60c2791928ebabf58129bce1beda0a83"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.788560 25971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:33.806406 25971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:33.808823 25971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:33.808938 25971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:33.812722 25971 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:37235
I20260925 10:17:33.812774 26038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:37235 every 8 connection(s)
I20260925 10:17:33.813138 25971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:33.813525 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 25971
I20260925 10:17:33.813644 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:33.813876 3683 external_mini_cluster.cc:1428] Setting key 9d3cb648b41cc8175c2001df300684b0
I20260925 10:17:33.814399 26039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:33.816453 26039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.152.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.253@KRBTEST.COM" created.
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:33.828308 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.817250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.3.152.254:37235)
0925 10:17:33.817446 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:33.817455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:33.817552 (+ 97us) server_negotiation.cc:407] Connection header received
0925 10:17:33.818104 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:33.818106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:33.818128 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:33.818226 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:33.818785 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.819597 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.820555 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.820871 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.821007 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.825039 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:33.825047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:33.825047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:33.825054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:33.826636 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.827073 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.827076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.827077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.827158 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.827381 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.827382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.827382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.827599 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:33.827693 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:33.827879 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 10:17:33.828045 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260925 10:17:33.828356 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.817216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37235 (local address 127.0.0.1:58006)
0925 10:17:33.817399 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.817446 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:33.817941 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:33.818234 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:33.818237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:33.818514 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:33.818674 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.818678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.819715 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.819716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.820454 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.820457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.820992 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.820994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.821057 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.821607 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:33.821615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:33.824921 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:33.826762 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.826769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.826771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.826964 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.827225 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.827226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.827227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.827315 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.827671 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:33.827676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:33.827739 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 10:17:33.827847 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:17:33.828125 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":116,"thread_start_us":90,"threads_started":1}
I20260925 10:17:33.829421 26039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:33.829793 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.829678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50728)
0925 10:17:33.829729 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.829773 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:17:33.829973 25988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
I20260925 10:17:33.830407 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.830319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50742)
0925 10:17:33.830366 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.830390 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:17:33.830551 26039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:32779: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
I20260925 10:17:33.839382 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:32779
--webserver_interface=127.3.152.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:17:33.879778 26039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } attempt: 1
I20260925 10:17:33.880347 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.880187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50758)
0925 10:17:33.880274 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.880316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 10:17:33.880683 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.880581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50774)
0925 10:17:33.880633 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.880659 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:17:33.880791 26039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:32779: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
W20260925 10:17:33.920533 26046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:33.920801 26046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:33.920831 26046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:33.920868 26046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:33.921979 26046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:33.922050 26046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:33.922076 26046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:33.922111 26046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:33.922137 26046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:33.923691 26046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.152.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:33.923969 26046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:33.924214 26046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:33.929131 26054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:33.929100 26052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:33.929090 26051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:33.929112 26046 server_base.cc:1034] running on GCE node
I20260925 10:17:33.929629 26046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:33.929939 26046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:33.931103 26046 hybrid_clock.cc:648] HybridClock initialized: now 1790331453931089 us; error 30 us; skew 500 ppm
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:33.934031 26046 init.cc:377] Logged in from keytab as kudu/127.3.152.253@KRBTEST.COM (short username kudu)
I20260925 10:17:33.934528 26046 webserver.cc:460] Webserver started at http://127.3.152.253:34649/ using document root <none> and password file <none>
I20260925 10:17:33.934782 26046 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:33.934854 26046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:33.935019 26046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:33.936568 26046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data/instance:
uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "a3338f962a805d75ce718fc25e83d45d"
server_key_iv: "89afe1a11ef2ba8fc414662914331d97"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.937017 26046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal/instance:
uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "a3338f962a805d75ce718fc25e83d45d"
server_key_iv: "89afe1a11ef2ba8fc414662914331d97"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.938786 26046 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 10:17:33.939780 26061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:33.940047 26046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:33.940136 26046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d"
format_stamp: "Formatted at 2026-09-25 10:17:33 on dist-test-slave-dzwq"
server_key: "a3338f962a805d75ce718fc25e83d45d"
server_key_iv: "89afe1a11ef2ba8fc414662914331d97"
server_key_version: "encryptionkey@0"
I20260925 10:17:33.940192 26046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:33.947443 26046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:33.948952 26039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } attempt: 2
I20260925 10:17:33.949376 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.949243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50786)
0925 10:17:33.949308 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.949345 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:17:33.949479 26046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:33.949707 26046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:33.949858 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.949727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50796)
0925 10:17:33.949795 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.949828 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:17:33.949966 26039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:32779: Network error: Client connection negotiation failed: client connection to 127.3.152.253:32779: connect: Connection refused (error 111)
I20260925 10:17:33.953842 26046 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.253:32779
I20260925 10:17:33.953907 26113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.253:32779 every 8 connection(s)
I20260925 10:17:33.954267 26046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
I20260925 10:17:33.955413 26114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:33.955703 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 26046
I20260925 10:17:33.955819 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-1/wal/instance
I20260925 10:17:33.956056 3683 external_mini_cluster.cc:1428] Setting key 8919a5bc00aa775fe45ba5e874a9fe77
I20260925 10:17:33.957964 26114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.152.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.252@KRBTEST.COM" created.
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:33.971944 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.958705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58016 (local address 127.3.152.254:37235)
0925 10:17:33.958777 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:33.958780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:33.959028 (+ 248us) server_negotiation.cc:407] Connection header received
0925 10:17:33.959781 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:33.959783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:33.959798 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:33.959892 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:33.960879 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.961377 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.962955 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.963184 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.963251 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.968688 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:33.968691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:33.968692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:33.968701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:33.970358 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.970889 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.970890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.970891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.970940 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.971240 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.971241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.971241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.971347 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:33.971398 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:33.971801 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 10:17:33.971875 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:17:33.972282 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.958672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37235 (local address 127.0.0.1:58016)
0925 10:17:33.958860 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.958910 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:33.959636 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:33.960035 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:33.960040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:33.960424 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:33.960753 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.960761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.961491 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.961492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.962845 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.962848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.963309 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.963311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.963468 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.964050 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:33.964061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:33.968582 (+ 4521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:33.970456 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.970464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.970467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.970811 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.971010 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.971011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.971011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.971173 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.971419 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:33.971429 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:33.971632 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 10:17:33.971777 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 10:17:33.971985 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260925 10:17:33.972951 26114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:17:33 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:33.981702 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.973233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50804)
0925 10:17:33.973283 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.973295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:33.973339 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:33.973614 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:33.973615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:33.973775 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:33.973842 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.973844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.974728 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.974730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.975580 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:33.975583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.976170 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.976171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:33.976254 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.976656 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:33.976660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:33.978794 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:33.980684 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.980684 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.980685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.980852 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.981099 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:33.981099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:33.981100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:33.981154 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:33.981428 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:33.981428 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:33.981466 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:17:33.981541 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:17:33.981624 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:17:33.981770 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.973310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50804 (local address 127.3.152.253:32779)
0925 10:17:33.973457 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:33.973462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:33.973483 (+ 21us) server_negotiation.cc:407] Connection header received
0925 10:17:33.973500 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:33.973502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:33.973512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:33.973580 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:33.973924 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.974627 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.975687 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:33.976084 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:33.976152 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:33.978937 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:33.978947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:33.978948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:33.978955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:33.980608 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.980926 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.980930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.980930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.981029 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:33.981210 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:33.981211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:33.981211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:33.981360 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:33.981452 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:33.981587 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 10:17:33.981675 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260925 10:17:33.982442 26114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:33.982779 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.982692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55878)
0925 10:17:33.982720 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.982762 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 10:17:33.983013 26064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
I20260925 10:17:33.983498 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:33.983400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55890)
0925 10:17:33.983449 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:33.983478 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:17:33.983733 26114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:45363: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
I20260925 10:17:33.985497 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:45363
--webserver_interface=127.3.152.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:17:34.033020 26114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } attempt: 1
I20260925 10:17:34.033619 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.033466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55904)
0925 10:17:34.033544 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.033589 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 10:17:34.034044 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.033926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55918)
0925 10:17:34.033979 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.034013 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 10:17:34.034173 26114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:45363: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
I20260925 10:17:34.041115 26039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } attempt: 3
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:34.048727 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.041560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50810)
0925 10:17:34.041627 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.041637 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.041702 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.041863 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.041864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.042029 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.042100 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.042102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.042906 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.042908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.043461 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.043463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.043902 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.043903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.044031 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.044525 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.044530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.045962 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.047598 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.047599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.047600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.047778 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.048041 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.048042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.048042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.048110 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.048369 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.048370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.048422 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.048510 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.048619 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:17:34.048725 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.041626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.3.152.253:32779)
0925 10:17:34.041704 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.041706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.041717 (+ 11us) server_negotiation.cc:407] Connection header received
0925 10:17:34.041768 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.041769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.041788 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.041856 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.042248 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.042811 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.043554 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.043820 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.043902 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.046062 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.046063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.046063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.046071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.047493 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.047867 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.047869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.047870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.047930 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.048175 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.048176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.048177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.048297 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.048377 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.048535 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.048619 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 10:17:34.049650 26039 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:34.050042 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.049918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55926)
0925 10:17:34.049978 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.050014 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:17:34.050405 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.050306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55936)
0925 10:17:34.050355 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.050384 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:17:34.050503 26039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:45363: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
W20260925 10:17:34.070796 26121 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:34.071028 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:34.071048 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:34.071120 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:34.072345 26121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:34.072389 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:34.072404 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:34.072417 26121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:34.072464 26121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:34.073957 26121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:37235,127.3.152.253:32779,127.3.152.252:45363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:45363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.152.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:34.074235 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:34.074481 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:34.079417 26126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:34.079385 26127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:34.079428 26129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:34.079814 26121 server_base.cc:1034] running on GCE node
I20260925 10:17:34.080106 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:34.080413 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:34.081589 26121 hybrid_clock.cc:648] HybridClock initialized: now 1790331454081580 us; error 41 us; skew 500 ppm
I20260925 10:17:34.081648 26039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } attempt: 1
I20260925 10:17:34.082070 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.081917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55950)
0925 10:17:34.081988 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.082027 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 10:17:34.082458 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.082345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55956)
0925 10:17:34.082399 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.082425 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:17:34.082610 26039 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:45363: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:34.084451 26121 init.cc:377] Logged in from keytab as kudu/127.3.152.252@KRBTEST.COM (short username kudu)
I20260925 10:17:34.084892 26121 webserver.cc:460] Webserver started at http://127.3.152.252:36989/ using document root <none> and password file <none>
I20260925 10:17:34.085143 26121 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:34.085223 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:34.085374 26121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:34.086462 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data/instance:
uuid: "9174b874697640889c3e24a2f5ac7c1b"
format_stamp: "Formatted at 2026-09-25 10:17:34 on dist-test-slave-dzwq"
server_key: "e2976dacd941919790596291dd906b07"
server_key_iv: "4b4a34e256095e689e8747e1d02abddd"
server_key_version: "encryptionkey@0"
I20260925 10:17:34.086836 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal/instance:
uuid: "9174b874697640889c3e24a2f5ac7c1b"
format_stamp: "Formatted at 2026-09-25 10:17:34 on dist-test-slave-dzwq"
server_key: "e2976dacd941919790596291dd906b07"
server_key_iv: "4b4a34e256095e689e8747e1d02abddd"
server_key_version: "encryptionkey@0"
I20260925 10:17:34.088238 26121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 10:17:34.088986 26136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:34.089174 26121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:34.089298 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "9174b874697640889c3e24a2f5ac7c1b"
format_stamp: "Formatted at 2026-09-25 10:17:34 on dist-test-slave-dzwq"
server_key: "e2976dacd941919790596291dd906b07"
server_key_iv: "4b4a34e256095e689e8747e1d02abddd"
server_key_version: "encryptionkey@0"
I20260925 10:17:34.089376 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:34.101460 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:34.102312 26114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } attempt: 2
I20260925 10:17:34.102757 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.102625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55970)
0925 10:17:34.102676 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.102720 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 10:17:34.103127 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.103037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:55984)
0925 10:17:34.103079 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.103105 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:17:34.103247 26114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:45363: Network error: Client connection negotiation failed: client connection to 127.3.152.252:45363: connect: Connection refused (error 111)
I20260925 10:17:34.103864 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:34.104056 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:34.107928 26121 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.252:45363
I20260925 10:17:34.108027 26188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.252:45363 every 8 connection(s)
I20260925 10:17:34.108376 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
I20260925 10:17:34.109433 26189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:34.110430 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 26121
I20260925 10:17:34.110526 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790331159020724-3683-0/minicluster-data/master-2/wal/instance
I20260925 10:17:34.110733 3683 external_mini_cluster.cc:1428] Setting key c8bd4786f36bbbbdba7348bbf7ba412d
I20260925 10:17:34.111265 26189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:34.120818 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.112006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.3.152.254:37235)
0925 10:17:34.112051 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.112053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.112190 (+ 137us) server_negotiation.cc:407] Connection header received
0925 10:17:34.112239 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.112240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.112251 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.112304 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.113080 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.113830 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.115006 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.115256 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.115324 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.117729 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.117731 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.117731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.117740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.119339 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.119763 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.119765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.119766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.119827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.120141 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.120142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.120143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.120293 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.120372 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.120630 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.120713 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 10:17:34.121001 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.111919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:37235 (local address 127.0.0.1:58026)
0925 10:17:34.112064 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.112102 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.112182 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.112327 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.112329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.112725 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.112971 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.112978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.113921 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.113922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.114917 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.114919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.115358 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.115359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.115462 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.115952 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.115998 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.117639 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.119399 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.119407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.119409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.119679 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.119898 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.119898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.119898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.120073 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.120367 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.120372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.120483 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.120591 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.120755 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260925 10:17:34.121949 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.111262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.3.152.252:45363)
0925 10:17:34.111461 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.111471 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.111519 (+ 48us) server_negotiation.cc:407] Connection header received
0925 10:17:34.112083 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.112092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.112156 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.112293 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.113079 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.114194 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.115183 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.115570 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.115622 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.118364 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.118371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.118371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.118378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.120386 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.120706 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.120709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.120709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.120770 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.120993 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.120994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.120994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.121176 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.121697 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.121762 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.121843 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260925 10:17:34.121981 26189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:34.129431 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.122256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:32779 (local address 127.0.0.1:50814)
0925 10:17:34.122364 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.122375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.122433 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.122579 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.122580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.122749 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.122798 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.122800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.123666 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.123667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.124352 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.124354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.124827 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.124828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.124893 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.125309 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.125314 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.126628 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.128436 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.128437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.128438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.128655 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.128867 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.128868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.128868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.128935 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.129171 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.129172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.129222 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.129292 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.129364 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 10:17:34.129560 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.122320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.3.152.253:32779)
0925 10:17:34.122364 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.122366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.122423 (+ 57us) server_negotiation.cc:407] Connection header received
0925 10:17:34.122479 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.122481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.122494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.122561 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.122874 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.123595 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.124427 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.124728 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.124807 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.126719 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.126720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.126720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.126727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.128354 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.128732 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.128733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.128734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.128788 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.129002 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.129002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.129002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.129107 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.129157 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.129345 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.129425 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:34.129864 26189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:34.130398 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.122760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.3.152.254:37235)
0925 10:17:34.122796 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.122797 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.122827 (+ 30us) server_negotiation.cc:407] Connection header received
0925 10:17:34.122847 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.122848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.122857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.122904 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.123427 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.124043 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.124826 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.125101 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.125182 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.127419 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.127420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.127420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.127427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.129125 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.129559 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.129560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.129561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.129621 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.129833 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.129834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.129835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.129969 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.130133 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.130198 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.130293 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:34.137025 26193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.130119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:56000)
0925 10:17:34.130182 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.130193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.130242 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.130622 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.130623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.130768 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.130793 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.130795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.131697 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.131698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.132270 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.132272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.132676 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.132677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.132724 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.133098 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.133102 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.134282 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.135929 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.135930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.135931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.136138 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.136341 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.136342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.136343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.136395 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.136672 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.136672 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.136709 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.136773 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.136884 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 10:17:34.137035 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.130197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.3.152.252:45363)
0925 10:17:34.130415 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.130416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.130424 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:34.130475 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.130476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.130554 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.130620 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.130870 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.131636 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.132360 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.132591 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.132676 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.134374 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.134376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.134377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.134385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.135831 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.136222 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.136223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.136223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.136277 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.136474 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.136475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.136475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.136596 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.136681 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.136809 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.136932 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":202}
I20260925 10:17:34.138592 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.130960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.3.152.253:32779)
0925 10:17:34.130993 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.130995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.131003 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:34.131063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.131063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.131071 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.131106 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.131625 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.132118 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.132787 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.133085 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.133159 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.135822 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.135824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.135824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.135831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.137445 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.137837 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.137838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.137838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.137893 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.138101 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.138102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.138102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.138206 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.138319 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.138426 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.138490 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 10:17:34.138882 26189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: Bootstrap starting.
I20260925 10:17:34.140005 26189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:34.140089 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:34.140324 26189 log.cc:824] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:34.141266 26189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: No bootstrap required, opened a new log
I20260925 10:17:34.141132 26013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aad22396b6e94086a82e060a3fc0c446"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58032
I20260925 10:17:34.143306 26189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.143469 26189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9174b874697640889c3e24a2f5ac7c1b, State: Initialized, Role: FOLLOWER
I20260925 10:17:34.143570 26189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.143957 26189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:34.144014 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.144172 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:17:34.145864 26206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:34.145928 26206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:34.146157 26121 master_runner.cc:186] Error getting master registration for 127.3.152.254:37235: OK, instance_id { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" instance_seqno: 1790331453811591 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:17:34.149071 26121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:17:34.157733 26039 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } attempt: 2
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:34.165892 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.158132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:56016)
0925 10:17:34.158194 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.158206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.158258 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.158415 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.158416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.158606 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.158654 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.158656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.159338 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.159340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.159893 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.159896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.160320 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.160321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.160384 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.160902 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.160908 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.162569 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.164648 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.164649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.164650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.164887 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.165117 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.165118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.165119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.165189 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.165550 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.165551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.165603 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.165678 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.165786 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:17:34.165911 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.158189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.3.152.252:45363)
0925 10:17:34.158256 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.158259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.158269 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:34.158318 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.158319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.158330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.158410 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.158758 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.159242 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.160016 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.160222 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.160318 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.162653 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.162654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.162655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.162663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.164554 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.164972 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.164973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.164974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.165042 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.165264 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.165265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.165265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.165472 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.165550 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.165716 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.165804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 10:17:34.167730 26039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: Bootstrap starting.
I20260925 10:17:34.168592 26039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:34.168900 26039 log.cc:824] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:34.169844 26039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: No bootstrap required, opened a new log
I20260925 10:17:34.171232 26039 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.171384 26039 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad22396b6e94086a82e060a3fc0c446, State: Initialized, Role: FOLLOWER
I20260925 10:17:34.171473 26039 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.171808 26039 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:34.171775 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.171875 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:34.171916 26013 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:17:34.172049 26013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:34.172915 26013 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.173552 26163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad22396b6e94086a82e060a3fc0c446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9174b874697640889c3e24a2f5ac7c1b"
I20260925 10:17:34.173707 26163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:34.175124 26163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad22396b6e94086a82e060a3fc0c446 in term 1.
I20260925 10:17:34.175289 26013 leader_election.cc:288] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [CANDIDATE]: Term 1 election: Requested vote from peers c9fa20f97b6742ebbd7ec2fa6f8e489d (127.3.152.253:32779), 9174b874697640889c3e24a2f5ac7c1b (127.3.152.252:45363)
I20260925 10:17:34.175419 25989 leader_election.cc:302] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9174b874697640889c3e24a2f5ac7c1b, aad22396b6e94086a82e060a3fc0c446; no voters:
W20260925 10:17:34.176039 26220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:34.176091 26220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:34.174703 25971 master_runner.cc:186] Error getting master registration for 127.3.152.253:32779: OK, instance_id { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" instance_seqno: 1790331453952594 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:17:34.177366 25971 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:17:34.177517 26209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:34.177675 26209 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [term 1 LEADER]: Becoming Leader. State: Replica: aad22396b6e94086a82e060a3fc0c446, State: Running, Role: LEADER
I20260925 10:17:34.177764 26209 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.179621 26222 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad22396b6e94086a82e060a3fc0c446. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.179862 26222 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:34.180439 26230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:34.180721 26230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:34.182466 26163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [term 1 FOLLOWER]: Refusing update from remote peer aad22396b6e94086a82e060a3fc0c446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:34.182866 26222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:34.184437 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad22396b6e94086a82e060a3fc0c446. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.184559 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:34.186430 26222 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.186549 26222 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:34.187136 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.187247 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:34.187670 26232 mvcc.cc:205] Tried to move back new op lower bound from 7333197636328472576 to 7333197636312854528. Current Snapshot: MvccSnapshot[applied={T|T < 7333197636328472576}]
I20260925 10:17:34.188764 26233 mvcc.cc:205] Tried to move back new op lower bound from 7333197636328472576 to 7333197636312854528. Current Snapshot: MvccSnapshot[applied={T|T < 7333197636328472576}]
I20260925 10:17:34.189373 26230 catalog_manager.cc:1347] Generated new cluster ID: ab46e952d5134079ad583df56f4c547e
I20260925 10:17:34.189430 26230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:34.193109 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.179797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.3.152.252:45363)
0925 10:17:34.179845 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.179847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.183046 (+ 3199us) server_negotiation.cc:407] Connection header received
0925 10:17:34.183154 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.183155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.183168 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.183203 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.184836 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.185526 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.186548 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.186819 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.186863 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.188633 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.188634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.188635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.188643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.191238 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.191949 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.191951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.191951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.192052 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.192387 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.192388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.192389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.192533 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.192622 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.192927 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.193013 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 10:17:34.194376 26114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } attempt: 3
I20260925 10:17:34.194779 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.179526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.3.152.254:37235)
0925 10:17:34.179596 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.179599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.180122 (+ 523us) server_negotiation.cc:407] Connection header received
0925 10:17:34.180243 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.180244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.180257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.180297 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.181220 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.181908 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.184088 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.184390 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.184443 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.188581 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.188583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.188583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.188591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.191955 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.193235 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.193237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.193237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.193298 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.193798 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.193799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.193799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.193946 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.194025 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.194603 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.194689 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 10:17:34 dist-test-slave-dzwq krb5kdc[25958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:34.202646 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.194641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:45363 (local address 127.0.0.1:56040)
0925 10:17:34.194684 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:34.194696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:34.194741 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:34.195029 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:34.195030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:34.195217 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:34.195270 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.195272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.196174 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.196175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.197002 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:34.197005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.197525 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.197526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:34.197587 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.198060 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:34.198065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:34.199593 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:34.201545 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.201547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.201547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.201765 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.201967 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:34.201968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:34.201968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:34.202026 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:34.202305 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:34.202306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:34.202351 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 10:17:34.202417 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 10:17:34.202533 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":3}
I20260925 10:17:34.203048 26192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.194847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.3.152.252:45363)
0925 10:17:34.194890 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.194892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.194900 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:34.194923 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.194923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.194935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.194984 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.195356 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.196077 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.197098 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.197403 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.197472 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.199694 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.199695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.199695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.199702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.201431 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.201842 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.201843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.201844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.201897 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.202097 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.202098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.202099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.202220 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.202280 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.202874 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.202952 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 10:17:34.204704 26114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: Bootstrap starting.
I20260925 10:17:34.205518 26114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:34.205856 26114 log.cc:824] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:34.206880 26114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: No bootstrap required, opened a new log
I20260925 10:17:34.207333 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:34.180304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.3.152.253:32779)
0925 10:17:34.180358 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.180360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.188206 (+ 7846us) server_negotiation.cc:407] Connection header received
0925 10:17:34.188324 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.188326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.188338 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.188370 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.200693 (+ 12323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.201387 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.202403 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:34.202724 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:34.202771 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:34.204332 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:34.204334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:34.204334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:34.204341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:34.206024 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.206424 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.206425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.206425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.206481 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:34.206710 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:34.206710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:34.206711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:34.206838 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:34.206938 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:34.207156 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 10:17:34.207240 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:17:34.209235 26114 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.209395 26114 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fa20f97b6742ebbd7ec2fa6f8e489d, State: Initialized, Role: FOLLOWER
I20260925 10:17:34.209497 26114 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } }
I20260925 10:17:34.209890 26114 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:34.209967 26088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aad22396b6e94086a82e060a3fc0c446" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d"
I20260925 10:17:34.209833 26238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.210039 26087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:34.211303 26087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [term 1 FOLLOWER]: Refusing update from remote peer aad22396b6e94086a82e060a3fc0c446: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:34.212121 26209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:34.212857 26238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:34.213528 26230 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:34.214092 26230 catalog_manager.cc:1504] Loading token signing keys...
I20260925 10:17:34.215814 26088 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad22396b6e94086a82e060a3fc0c446 in term 1.
I20260925 10:17:34.217727 26251 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.218387 26251 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:17:34.218331 26252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:34.218520 26252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:34.217589 26238 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad22396b6e94086a82e060a3fc0c446. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.219736 26238 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:34.220881 26209 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aad22396b6e94086a82e060a3fc0c446" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad22396b6e94086a82e060a3fc0c446" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 37235 } } peers { permanent_uuid: "c9fa20f97b6742ebbd7ec2fa6f8e489d" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 32779 } } peers { permanent_uuid: "9174b874697640889c3e24a2f5ac7c1b" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 45363 } } }
I20260925 10:17:34.221000 26209 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:34.221325 26230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad22396b6e94086a82e060a3fc0c446: Generated new TSK 0
I20260925 10:17:34.221546 26230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:34.222374 26249 mvcc.cc:205] Tried to move back new op lower bound from 7333197636449370112 to 7333197636312854528. Current Snapshot: MvccSnapshot[applied={T|T < 7333197636328472576}]
W20260925 10:17:34.248062 26192 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.246233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.3.152.252:45363)
0925 10:17:34.246282 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.246284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.247287 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 10:17:34.247410 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.247411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.247424 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.247459 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.247941 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56044: BlockingRecv error: recv got EOF from 127.0.0.1:56044 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 10:17:34.249076 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.246639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.3.152.253:32779)
0925 10:17:34.246687 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.246689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.248497 (+ 1808us) server_negotiation.cc:407] Connection header received
0925 10:17:34.248613 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.248614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.248626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.248660 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.249009 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50846: BlockingRecv error: recv got EOF from 127.0.0.1:50846 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 10:17:34.250005 26043 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.247016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.3.152.254:37235)
0925 10:17:34.247071 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.247073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.249459 (+ 2386us) server_negotiation.cc:407] Connection header received
0925 10:17:34.249543 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.249544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.249557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.249649 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.249927 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58048: BlockingRecv error: recv got EOF from 127.0.0.1:58048 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 10:17:34.252316 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.251457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50858 (local address 127.3.152.253:32779)
0925 10:17:34.251496 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.251498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.251710 (+ 212us) server_negotiation.cc:407] Connection header received
0925 10:17:34.251826 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.251827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.251845 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.251878 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.252269 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50858: BlockingRecv error: recv got EOF from 127.0.0.1:50858 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 10:17:34.255502 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.254703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.3.152.253:32779)
0925 10:17:34.254738 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.254740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.254982 (+ 242us) server_negotiation.cc:407] Connection header received
0925 10:17:34.255087 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.255088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.255099 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.255130 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.255452 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50870: BlockingRecv error: recv got EOF from 127.0.0.1:50870 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 10:17:34.259655 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.258835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.3.152.253:32779)
0925 10:17:34.258878 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.258880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.259129 (+ 249us) server_negotiation.cc:407] Connection header received
0925 10:17:34.259235 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.259236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.259247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.259279 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.259606 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50880: BlockingRecv error: recv got EOF from 127.0.0.1:50880 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 10:17:34.265928 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.265077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50890 (local address 127.3.152.253:32779)
0925 10:17:34.265119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.265121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.265366 (+ 245us) server_negotiation.cc:407] Connection header received
0925 10:17:34.265477 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.265478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.265489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.265523 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.265873 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50890: BlockingRecv error: recv got EOF from 127.0.0.1:50890 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:17:34.276464 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.275457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.3.152.253:32779)
0925 10:17:34.275504 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.275506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.275787 (+ 281us) server_negotiation.cc:407] Connection header received
0925 10:17:34.275918 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.275919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.275931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.275999 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.276392 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50894: BlockingRecv error: recv got EOF from 127.0.0.1:50894 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:17:34.295146 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.294098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.3.152.253:32779)
0925 10:17:34.294160 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.294163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.294453 (+ 290us) server_negotiation.cc:407] Connection header received
0925 10:17:34.294578 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.294580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.294593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.294628 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.295078 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50910: BlockingRecv error: recv got EOF from 127.0.0.1:50910 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:17:34.329849 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.328856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.3.152.253:32779)
0925 10:17:34.328903 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.328905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.329213 (+ 308us) server_negotiation.cc:407] Connection header received
0925 10:17:34.329323 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.329324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.329336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.329367 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.329785 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50922: BlockingRecv error: recv got EOF from 127.0.0.1:50922 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:17:34.396332 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.395555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.3.152.253:32779)
0925 10:17:34.395640 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.395643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.395773 (+ 130us) server_negotiation.cc:407] Connection header received
0925 10:17:34.395823 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.395824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.395835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.395894 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.396257 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50930: BlockingRecv error: recv got EOF from 127.0.0.1:50930 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 10:17:34.526368 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.525650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.3.152.253:32779)
0925 10:17:34.525725 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.525727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.525766 (+ 39us) server_negotiation.cc:407] Connection header received
0925 10:17:34.525871 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.525872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.525888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.525981 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.526313 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50942: BlockingRecv error: recv got EOF from 127.0.0.1:50942 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 10:17:34.784461 26119 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:34.783852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.3.152.253:32779)
0925 10:17:34.783924 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:34.783926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:34.783936 (+ 10us) server_negotiation.cc:407] Connection header received
0925 10:17:34.784026 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:34.784027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:34.784038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:34.784086 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:34.784409 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50956: BlockingRecv error: recv got EOF from 127.0.0.1:50956 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:17:35.146436 26206 catalog_manager.cc:1259] Loaded cluster ID: ab46e952d5134079ad583df56f4c547e
I20260925 10:17:35.146528 26206 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: loading cluster ID for follower catalog manager: success
I20260925 10:17:35.147300 26206 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: acquiring CA information for follower catalog manager: success
I20260925 10:17:35.147621 26206 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9174b874697640889c3e24a2f5ac7c1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:17:35.219116 26252 catalog_manager.cc:1259] Loaded cluster ID: ab46e952d5134079ad583df56f4c547e
I20260925 10:17:35.219202 26252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: loading cluster ID for follower catalog manager: success
I20260925 10:17:35.220103 26252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: acquiring CA information for follower catalog manager: success
I20260925 10:17:35.220394 26252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c9fa20f97b6742ebbd7ec2fa6f8e489d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:17:35.300518 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:35.297789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.3.152.253:32779)
0925 10:17:35.297914 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:35.297916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:35.297925 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:35.297952 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:35.297953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:35.298012 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:35.298102 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:35.298240 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.298793 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.299601 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.299808 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.299875 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:35.300139 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:35.300266 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:35.300346 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:35.300383 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 10:17:35.300415 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260925 10:17:35.305001 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:35.302566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.3.152.252:45363)
0925 10:17:35.302714 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:35.302716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:35.302725 (+ 9us) server_negotiation.cc:407] Connection header received
0925 10:17:35.302767 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:35.302767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:35.302804 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:35.302888 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:35.302993 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.303549 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.304281 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.304496 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.304557 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:35.304690 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:35.304802 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:35.304867 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:35.304912 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 10:17:35.304931 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260925 10:17:35.310108 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:35.306818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.3.152.253:32779)
0925 10:17:35.306872 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:35.306873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:35.307000 (+ 127us) server_negotiation.cc:407] Connection header received
0925 10:17:35.307116 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:35.307118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:35.307133 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:35.307194 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:35.307393 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.308039 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309144 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.309442 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309508 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:35.309710 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:35.309751 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:35.309799 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:35.309952 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 10:17:35.310002 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 10:17:35.310216 26329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:35.307013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.3.152.252:45363)
0925 10:17:35.307060 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:35.307061 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:35.307337 (+ 276us) server_negotiation.cc:407] Connection header received
0925 10:17:35.307386 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:35.307387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:35.307396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:35.307444 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:35.307625 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.308373 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309277 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.309555 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309589 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:35.309776 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:35.309812 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:35.309873 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:35.310093 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 10:17:35.310133 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:17:35.310789 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:35.306886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.3.152.254:37235)
0925 10:17:35.307044 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:35.307046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:35.307151 (+ 105us) server_negotiation.cc:407] Connection header received
0925 10:17:35.307249 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:35.307250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:35.307288 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:35.307390 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:35.307529 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.308366 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309434 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:35.309809 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:35.309881 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:35.310314 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:35.310442 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:35.310509 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:35.310604 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 10:17:35.310634 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260925 10:17:35.312750 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 25971
I20260925 10:17:35.318784 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 26046
I20260925 10:17:35.323812 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 26121
2026-09-25T10:17:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2790 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 10:17:35 dist-test-slave-dzwq krb5kdc[26340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 10:17:35 dist-test-slave-dzwq krb5kdc[26340](info): set up 2 sockets
Sep 25 10:17:35 dist-test-slave-dzwq krb5kdc[26340](info): commencing operation
krb5kdc: starting...
W20260925 10:17:36.448939 3683 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/test-user.keytab.
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T10:17:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T10:17:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.152.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
Entry for principal kudu/127.3.152.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab.
I20260925 10:17:36.580336 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:39161
--webserver_interface=127.3.152.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 10:17:36.662420 26353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:36.662693 26353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:36.662725 26353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:36.662760 26353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:36.663913 26353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:36.664011 26353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:36.664048 26353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:36.664070 26353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:36.664100 26353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:36.665616 26353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.254.keytab
--principal=kudu/127.3.152.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.254:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.152.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:36.665902 26353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:36.666134 26353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:36.671214 26358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:36.671232 26359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:36.671214 26361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:36.671308 26353 server_base.cc:1034] running on GCE node
I20260925 10:17:36.671658 26353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:36.672058 26353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:36.673238 26353 hybrid_clock.cc:648] HybridClock initialized: now 1790331456673216 us; error 38 us; skew 500 ppm
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:36.679661 26353 init.cc:377] Logged in from keytab as kudu/127.3.152.254@KRBTEST.COM (short username kudu)
I20260925 10:17:36.680212 26353 webserver.cc:460] Webserver started at http://127.3.152.254:39589/ using document root <none> and password file <none>
I20260925 10:17:36.680406 26353 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:36.680449 26353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:36.680540 26353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:36.681613 26353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data/instance:
uuid: "0f65d5b04d0049e8809b9a0bcd993688"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "086b73558135fbcfcbc60a3a9334839f"
server_key_iv: "75c4dda5579d1f53095855d1c45f369b"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.681937 26353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance:
uuid: "0f65d5b04d0049e8809b9a0bcd993688"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "086b73558135fbcfcbc60a3a9334839f"
server_key_iv: "75c4dda5579d1f53095855d1c45f369b"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.683254 26353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:36.683943 26368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:36.684123 26353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 10:17:36.684228 26353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal
uuid: "0f65d5b04d0049e8809b9a0bcd993688"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "086b73558135fbcfcbc60a3a9334839f"
server_key_iv: "75c4dda5579d1f53095855d1c45f369b"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.684286 26353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:36.699338 26353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:36.702284 26353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:36.702485 26353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:36.706881 26353 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.254:39161
I20260925 10:17:36.706911 26420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.254:39161 every 8 connection(s)
I20260925 10:17:36.707296 26353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/data/info.pb
I20260925 10:17:36.708416 26421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:36.710032 26421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:36.716300 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 26353
I20260925 10:17:36.716394 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-0/wal/instance
I20260925 10:17:36.716607 3683 external_mini_cluster.cc:1428] Setting key 2241597fab1fd1e5e1ec2010b91ea9b5
I20260925 10:17:36.720184 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.710853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.3.152.254:39161)
0925 10:17:36.711081 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:36.711085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:36.711114 (+ 29us) server_negotiation.cc:407] Connection header received
0925 10:17:36.711640 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:36.711642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:36.711659 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:36.711737 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:36.712381 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.713227 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.714082 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.714389 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.714524 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.716854 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:36.716862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:36.716862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:36.716868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:36.718474 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.718950 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.718952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.718952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.719035 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.719288 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.719288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.719289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.719492 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:36.719556 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:36.719794 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 10:17:36.719906 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260925 10:17:36.720212 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.710818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39161 (local address 127.0.0.1:38282)
0925 10:17:36.711006 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.711042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:36.711498 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:36.711767 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:36.711769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:36.712083 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:36.712260 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.712264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.713313 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.713314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.713997 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.713999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.714483 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.714484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.714537 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.715039 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:36.715052 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:36.716769 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:36.718568 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.718575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.718577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.718867 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.719105 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.719106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.719106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.719220 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.719564 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:36.719571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:36.719648 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 10:17:36.719772 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 10:17:36.720024 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":69,"thread_start_us":99,"threads_started":1}
I20260925 10:17:36.721026 26421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:36.721419 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.721275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60506)
0925 10:17:36.721330 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.721385 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:17:36.721576 26372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
I20260925 10:17:36.722059 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.721948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60512)
0925 10:17:36.721997 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.722028 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:17:36.722179 26421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:37061: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.152.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
Entry for principal kudu/127.3.152.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab.
I20260925 10:17:36.743623 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:37061
--webserver_interface=127.3.152.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:17:36.771430 26421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } attempt: 1
I20260925 10:17:36.772004 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.771813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60528)
0925 10:17:36.771889 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.771942 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 10:17:36.772418 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.772315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60544)
0925 10:17:36.772363 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.772395 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:17:36.772549 26421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:37061: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
W20260925 10:17:36.826191 26428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:36.826432 26428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:36.826449 26428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:36.826584 26428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:36.827787 26428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:36.827836 26428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:36.827852 26428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:36.827864 26428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:36.827876 26428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:36.829517 26428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.253.keytab
--principal=kudu/127.3.152.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.253:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.152.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:36.829803 26428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:36.830098 26428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:36.835201 26436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:36.835175 26434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:36.835175 26433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:36.835424 26428 server_base.cc:1034] running on GCE node
I20260925 10:17:36.835680 26428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:36.835937 26428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:36.837148 26428 hybrid_clock.cc:648] HybridClock initialized: now 1790331456837142 us; error 36 us; skew 500 ppm
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:36.839775 26428 init.cc:377] Logged in from keytab as kudu/127.3.152.253@KRBTEST.COM (short username kudu)
I20260925 10:17:36.840317 26428 webserver.cc:460] Webserver started at http://127.3.152.253:34681/ using document root <none> and password file <none>
I20260925 10:17:36.840557 26428 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:36.840628 26428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:36.840695 26421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } attempt: 2
I20260925 10:17:36.840788 26428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:36.841153 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.840999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60552)
0925 10:17:36.841070 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.841111 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 10:17:36.841581 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.841427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60564)
0925 10:17:36.841528 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.841559 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260925 10:17:36.841711 26421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.253:37061: Network error: Client connection negotiation failed: client connection to 127.3.152.253:37061: connect: Connection refused (error 111)
I20260925 10:17:36.842159 26428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data/instance:
uuid: "facadad8eefa42f79b75ec04a399cbe1"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "56d7556c761773c639d13174db070206"
server_key_iv: "eb6cf76a41bc3cc95610e6a9fd93ecdd"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.842542 26428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal/instance:
uuid: "facadad8eefa42f79b75ec04a399cbe1"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "56d7556c761773c639d13174db070206"
server_key_iv: "eb6cf76a41bc3cc95610e6a9fd93ecdd"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.843927 26428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 10:17:36.844758 26443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:36.844946 26428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:36.845047 26428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal
uuid: "facadad8eefa42f79b75ec04a399cbe1"
format_stamp: "Formatted at 2026-09-25 10:17:36 on dist-test-slave-dzwq"
server_key: "56d7556c761773c639d13174db070206"
server_key_iv: "eb6cf76a41bc3cc95610e6a9fd93ecdd"
server_key_version: "encryptionkey@0"
I20260925 10:17:36.845139 26428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:36.869065 26428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:36.871069 26428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:36.871268 26428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:36.875409 26428 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.253:37061
I20260925 10:17:36.875447 26495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.253:37061 every 8 connection(s)
I20260925 10:17:36.875841 26428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/data/info.pb
I20260925 10:17:36.876857 26496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:36.878456 26496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:36.878896 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 26428
I20260925 10:17:36.879012 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-1/wal/instance
I20260925 10:17:36.879253 3683 external_mini_cluster.cc:1428] Setting key 7cfd7f465c3d59ec13fb1b5ef12d282c
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.152.252@KRBTEST.COM; defaulting to no policy
I20260925 10:17:36.888437 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.879215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.3.152.254:39161)
0925 10:17:36.879294 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:36.879297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:36.879516 (+ 219us) server_negotiation.cc:407] Connection header received
0925 10:17:36.879978 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:36.879979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:36.879993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:36.880072 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:36.880803 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.881295 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.882418 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.882789 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.882876 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.885345 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:36.885347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:36.885347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:36.885356 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:36.887047 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.887485 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.887486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.887487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.887550 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.887786 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.887787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.887787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.887897 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:36.887950 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:36.888289 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 10:17:36.888368 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 10:17:36.888748 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.879193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39161 (local address 127.0.0.1:38298)
0925 10:17:36.879370 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.879410 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:36.879866 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:36.880128 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:36.880131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:36.880388 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:36.880698 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.880705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.881404 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.881405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.882314 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.882317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.882899 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.882901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.883010 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.883549 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:36.883565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:36.885243 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:36.887136 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.887143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.887145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.887427 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.887622 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.887623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.887623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.887726 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.887957 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:36.888003 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:36.888137 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 10:17:36.888268 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 10:17:36.888461 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260925 10:17:36.889400 26496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.152.252@KRBTEST.COM" created.
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:36.897617 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.889701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60572)
0925 10:17:36.889758 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.889770 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:36.889813 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:36.890188 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:36.890189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:36.890340 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:36.890402 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.890404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.891248 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.891249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.891880 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.891882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.892595 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.892596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.892679 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.893096 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:36.893101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:36.894472 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:36.896579 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.896579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.896580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.896737 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.896973 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.896973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.896974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.897025 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.897377 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:36.897377 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:36.897412 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 10:17:36.897462 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 10:17:36.897548 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 10:17:36.897761 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.889815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.3.152.253:37061)
0925 10:17:36.889981 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:36.889986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:36.890008 (+ 22us) server_negotiation.cc:407] Connection header received
0925 10:17:36.890035 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:36.890037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:36.890047 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:36.890138 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:36.890489 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.891163 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.892010 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.892511 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.892579 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.894564 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:36.894575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:36.894575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:36.894583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:36.896496 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.896828 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.896832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.896832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.896900 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.897088 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.897089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.897089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.897306 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:36.897400 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:36.897505 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:17:36.897620 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260925 10:17:36.898773 26496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:36.899155 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.899031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49840)
0925 10:17:36.899083 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.899133 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 10:17:36.899309 26446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
I20260925 10:17:36.899753 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.899665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49844)
0925 10:17:36.899712 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.899736 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 10:17:36.899891 26496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:37085: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
Entry for principal kudu/127.3.152.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab.
I20260925 10:17:36.907542 3683 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcY4CcD/build/release/bin/kudu
/tmp/dist-test-taskcY4CcD/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:37085
--webserver_interface=127.3.152.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.152.212:38313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 10:17:36.932888 26421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } attempt: 3
Sep 25 10:17:36 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:36.940761 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.933260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60586)
0925 10:17:36.933325 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.933339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:36.933392 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:36.933706 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:36.933707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:36.933880 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:36.933952 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.933953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.934919 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.934920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.935433 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:36.935434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.936033 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.936034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:36.936105 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.936600 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:36.936604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:36.937833 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:36.939725 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.939726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.939727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.939891 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.940181 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:36.940181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:36.940182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:36.940237 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:36.940518 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:36.940519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:36.940557 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 10:17:36.940608 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 10:17:36.940689 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 10:17:36.940836 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.933360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.3.152.253:37061)
0925 10:17:36.933433 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:36.933435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:36.933448 (+ 13us) server_negotiation.cc:407] Connection header received
0925 10:17:36.933483 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:36.933485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:36.933501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:36.933596 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:36.934046 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.934826 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.935525 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:36.935879 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:36.936002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:36.937924 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:36.937925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:36.937926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:36.937935 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:36.939628 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.940031 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.940032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.940033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.940104 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:36.940306 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:36.940307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:36.940307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:36.940450 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:36.940532 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:36.940637 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 10:17:36.940725 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 10:17:36.941282 26421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:36.941653 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.941545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49854)
0925 10:17:36.941596 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.941628 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 10:17:36.942018 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.941938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49858)
0925 10:17:36.941980 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.942003 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 10:17:36.942137 26421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:37085: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
I20260925 10:17:36.949175 26496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } attempt: 1
I20260925 10:17:36.949623 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.949504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49862)
0925 10:17:36.949559 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.949594 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 10:17:36.950016 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.949931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49868)
0925 10:17:36.949973 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.949998 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 10:17:36.950232 26496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:37085: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
I20260925 10:17:36.973330 26421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } attempt: 1
I20260925 10:17:36.973932 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.973753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49878)
0925 10:17:36.973831 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.973888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 10:17:36.974359 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:36.974247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49892)
0925 10:17:36.974300 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:36.974332 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 10:17:36.974491 26421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:37085: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
W20260925 10:17:36.995414 26503 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 10:17:36.995705 26503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 10:17:36.995724 26503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 10:17:36.995745 26503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 10:17:36.997187 26503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 10:17:36.997263 26503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 10:17:36.997279 26503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 10:17:36.997292 26503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 10:17:36.997304 26503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 10:17:36.998907 26503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.152.212:38313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.152.254:39161,127.3.152.253:37061,127.3.152.252:37085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/krb5kdc/kudu_127.3.152.252.keytab
--principal=kudu/127.3.152.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.152.252:37085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.152.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:44:32 UTC on ccycloud.quasar-afbmdh.root.comops.site
build id 4671575
I20260925 10:17:36.999208 26503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 10:17:36.999477 26503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 10:17:37.004451 26511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 10:17:37.004477 26508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:37.004756 26503 server_base.cc:1034] running on GCE node
W20260925 10:17:37.004469 26509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 10:17:37.005927 26503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 10:17:37.006243 26503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 10:17:37.007419 26503 hybrid_clock.cc:648] HybridClock initialized: now 1790331457007396 us; error 38 us; skew 500 ppm
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331457, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 10:17:37.010164 26503 init.cc:377] Logged in from keytab as kudu/127.3.152.252@KRBTEST.COM (short username kudu)
I20260925 10:17:37.010668 26503 webserver.cc:460] Webserver started at http://127.3.152.252:34109/ using document root <none> and password file <none>
I20260925 10:17:37.010926 26503 fs_manager.cc:359] Metadata directory not provided
I20260925 10:17:37.010999 26503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 10:17:37.011157 26503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 10:17:37.012380 26503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data/instance:
uuid: "9fd8295cd69b42aa95d523a60bbf7011"
format_stamp: "Formatted at 2026-09-25 10:17:37 on dist-test-slave-dzwq"
server_key: "dfe64880b01d071d0019ce09a9b3ad43"
server_key_iv: "371ff1687ccd53c989a811c9f23dc1ad"
server_key_version: "encryptionkey@0"
I20260925 10:17:37.012776 26503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal/instance:
uuid: "9fd8295cd69b42aa95d523a60bbf7011"
format_stamp: "Formatted at 2026-09-25 10:17:37 on dist-test-slave-dzwq"
server_key: "dfe64880b01d071d0019ce09a9b3ad43"
server_key_iv: "371ff1687ccd53c989a811c9f23dc1ad"
server_key_version: "encryptionkey@0"
I20260925 10:17:37.014191 26503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 10:17:37.015018 26518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 10:17:37.015196 26503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 10:17:37.015300 26503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data,/tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal
uuid: "9fd8295cd69b42aa95d523a60bbf7011"
format_stamp: "Formatted at 2026-09-25 10:17:37 on dist-test-slave-dzwq"
server_key: "dfe64880b01d071d0019ce09a9b3ad43"
server_key_iv: "371ff1687ccd53c989a811c9f23dc1ad"
server_key_version: "encryptionkey@0"
I20260925 10:17:37.015383 26503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 10:17:37.018392 26496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } attempt: 2
I20260925 10:17:37.018831 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.018675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49908)
0925 10:17:37.018754 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.018793 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 10:17:37.019202 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.019081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49924)
0925 10:17:37.019139 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.019177 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 10:17:37.019345 26496 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.152.252:37085: Network error: Client connection negotiation failed: client connection to 127.3.152.252:37085: connect: Connection refused (error 111)
I20260925 10:17:37.031507 26503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 10:17:37.034121 26503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 10:17:37.034248 26503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 10:17:37.038368 26503 rpc_server.cc:307] RPC server started. Bound to: 127.3.152.252:37085
I20260925 10:17:37.038430 26570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.152.252:37085 every 8 connection(s)
I20260925 10:17:37.038797 26503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/data/info.pb
I20260925 10:17:37.039867 26571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 10:17:37.041596 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:37.042721 3683 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcY4CcD/build/release/bin/kudu as pid 26503
I20260925 10:17:37.042855 3683 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcY4CcD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790331159020724-3683-0/minicluster-data/master-2/wal/instance
I20260925 10:17:37.043087 3683 external_mini_cluster.cc:1428] Setting key f5cc62aa9a372d372a33e42383998769
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331457, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:37.049675 26421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } attempt: 2
I20260925 10:17:37.052214 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.042383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.3.152.254:39161)
0925 10:17:37.042552 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.042555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.042718 (+ 163us) server_negotiation.cc:407] Connection header received
0925 10:17:37.043358 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.043359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.043388 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.043476 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.044376 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.044942 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.045969 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.046181 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.046279 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.048914 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.048916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.048916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.048923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.050604 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.051073 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.051074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.051074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.051139 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.051435 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.051436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.051437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.051576 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.051648 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.051999 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.052095 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 10:17:37.052342 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.042363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.254:39161 (local address 127.0.0.1:38304)
0925 10:17:37.042575 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.042624 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:37.043222 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:37.043575 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:37.043579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:37.044013 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:37.044268 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.044276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.045049 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.045050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.045847 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.045849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.046313 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.046315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.046442 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.047034 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:37.047046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:37.048786 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:37.050684 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.050688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.050690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.050990 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.051212 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.051213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.051214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.051353 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.051647 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:37.051651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:37.051773 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 10:17:37.051926 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 10:17:37.052106 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":96,"threads_started":1}
I20260925 10:17:37.053179 26575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.043503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.3.152.252:37085)
0925 10:17:37.043664 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.043668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.043691 (+ 23us) server_negotiation.cc:407] Connection header received
0925 10:17:37.043738 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.043740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.043760 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.043833 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.044626 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.045430 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.046177 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.046499 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.046584 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.049374 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.049383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.049383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.049390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.051361 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.052040 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.052043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.052044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.052107 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.052480 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.052481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.052481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.052634 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.052745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.052879 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.052957 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260925 10:17:37.053437 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.254@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:37.058874 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.050245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.3.152.252:37085)
0925 10:17:37.050715 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.050716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.050722 (+ 6us) server_negotiation.cc:407] Connection header received
0925 10:17:37.050737 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.050738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.050746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.050828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.051116 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.051799 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.052800 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.053079 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.053152 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.055398 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.055399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.055400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.055406 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.057741 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.058149 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.058150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.058150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.058208 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.058422 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.058422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.058423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.058518 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.058566 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.058730 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.058786 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":79,"threads_started":1}
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331457, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:37.058912 26423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.050176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49940)
0925 10:17:37.050466 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.050477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:37.050530 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:37.050833 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:37.050834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:37.050972 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:37.051010 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.051012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.051885 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.051887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.052717 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.052720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.053164 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.053165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.053231 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.053796 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:37.053802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:37.055309 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:37.057839 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.057840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.057841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.058062 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.058286 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.058287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.058287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.058348 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.058570 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:37.058571 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:37.058623 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 10:17:37.058716 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 10:17:37.058813 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.254@KRBTEST.COM
I20260925 10:17:37.060791 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: Bootstrap starting.
I20260925 10:17:37.061631 26421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:37.061990 26421 log.cc:824] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:37.063072 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.054096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.3.152.254:39161)
0925 10:17:37.054153 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.054155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.054163 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:37.054181 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.054182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.054190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.054247 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.054828 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.055561 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.056591 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.056802 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.057134 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.060046 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.060047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.060048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.060054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.061694 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062177 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.062178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.062178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.062243 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062486 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.062488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.062488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.062614 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.062800 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.062929 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.063013 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 10:17:37.063210 26421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: No bootstrap required, opened a new log
I20260925 10:17:37.063521 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.053771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.3.152.253:37061)
0925 10:17:37.053826 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.053828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.054455 (+ 627us) server_negotiation.cc:407] Connection header received
0925 10:17:37.054573 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.054574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.054585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.054615 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.054972 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.055677 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.056805 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.057120 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.057169 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.060121 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.060122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.060123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.060129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.062341 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062667 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.062669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.062669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.062729 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062936 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.062937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.062937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.063067 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.063138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.063343 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.063426 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 10:17:37.063796 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.053717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.253:37061 (local address 127.0.0.1:60602)
0925 10:17:37.054391 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.054401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:37.054485 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:37.054683 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:37.054684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:37.054829 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:37.054884 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.054886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.055763 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.055764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.056696 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.056698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.057269 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.057270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.057348 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.057787 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:37.057792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:37.060048 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:37.062423 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.062424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.062596 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.062798 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.062798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.062799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.062867 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.063171 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:37.063172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:37.063216 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 10:17:37.063607 (+ 391us) client_negotiation.cc:240] Negotiation successful
0925 10:17:37.063704 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653}
I20260925 10:17:37.064184 26571 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } has no permanent_uuid. Determining permanent_uuid...
I20260925 10:17:37.065588 26421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.065753 26421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f65d5b04d0049e8809b9a0bcd993688, State: Initialized, Role: FOLLOWER
I20260925 10:17:37.065853 26421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.066171 26578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.066237 26421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:37.066340 26578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:17:37.068329 26589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:37.068383 26589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331457, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.252@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.152.253@KRBTEST.COM
I20260925 10:17:37.069199 26353 master_runner.cc:186] Error getting master registration for 127.3.152.253:37061: OK, instance_id { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" instance_seqno: 1790331456874252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:17:37.071949 26353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:17:37.073000 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.064517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.3.152.252:37085)
0925 10:17:37.064658 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.064660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.064665 (+ 5us) server_negotiation.cc:407] Connection header received
0925 10:17:37.064682 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.064683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.064691 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.064740 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.065038 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.065726 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.066646 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.066972 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.067036 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.069688 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.069689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.069689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.069696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.071714 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.072119 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.072120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.072120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.072173 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.072342 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.072343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.072343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.072544 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.072616 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.072807 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.072893 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260925 10:17:37.073060 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.064483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49948)
0925 10:17:37.064550 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.064560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:37.064597 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:37.064763 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:37.064764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:37.064907 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:37.064949 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.064950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.065804 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.065805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.066554 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.066556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.067082 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.067082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.067129 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.068086 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:37.068091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:37.069595 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:37.071803 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.071804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.071804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.072043 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.072233 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.072234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.072234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.072284 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.072632 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:37.072633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:37.072691 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 10:17:37.072783 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 10:17:37.072944 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":71}
I20260925 10:17:37.074674 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: Bootstrap starting.
I20260925 10:17:37.075443 26571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:37.075793 26571 log.cc:824] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:37.076105 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.064111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.3.152.253:37061)
0925 10:17:37.064143 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.064145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.064152 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:17:37.064171 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.064172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.064181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.064236 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.064853 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.065614 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.066684 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.066984 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.067076 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.072432 (+ 5356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.072434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.072434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.072441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.074567 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.075022 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.075024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.075025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.075095 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.075330 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.075331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.075331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.075477 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.075663 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.075895 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.076009 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:17:37.076798 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: No bootstrap required, opened a new log
I20260925 10:17:37.077809 3683 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 10:17:37.078713 26571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.078886 26571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd8295cd69b42aa95d523a60bbf7011, State: Initialized, Role: FOLLOWER
I20260925 10:17:37.078842 26395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0f65d5b04d0049e8809b9a0bcd993688"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38310
I20260925 10:17:37.078981 26395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 10:17:37.078969 26571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.079034 26395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:37.079314 26571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:37.079280 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.079378 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:37.080085 26395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.080390 26395 leader_election.cc:288] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [CANDIDATE]: Term 1 election: Requested vote from peers facadad8eefa42f79b75ec04a399cbe1 (127.3.152.253:37061), 9fd8295cd69b42aa95d523a60bbf7011 (127.3.152.252:37085)
I20260925 10:17:37.080734 26545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f65d5b04d0049e8809b9a0bcd993688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9fd8295cd69b42aa95d523a60bbf7011"
I20260925 10:17:37.081113 26545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [term 0 FOLLOWER]: Advancing to term 1
W20260925 10:17:37.081825 26603 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:37.081902 26603 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:37.082059 26545 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f65d5b04d0049e8809b9a0bcd993688 in term 1.
I20260925 10:17:37.082295 26371 leader_election.cc:302] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f65d5b04d0049e8809b9a0bcd993688, 9fd8295cd69b42aa95d523a60bbf7011; no voters:
I20260925 10:17:37.082494 26578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 10:17:37.082681 26578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [term 1 LEADER]: Becoming Leader. State: Replica: 0f65d5b04d0049e8809b9a0bcd993688, State: Running, Role: LEADER
I20260925 10:17:37.082785 26578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.083262 26578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f65d5b04d0049e8809b9a0bcd993688. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.083355 26578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:37.084589 26503 master_runner.cc:186] Error getting master registration for 127.3.152.253:37061: OK, instance_id { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" instance_seqno: 1790331456874252 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 10:17:37.087139 26503 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 10:17:37.092119 26612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 10:17:37.092422 26612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 10:17:37.094287 26500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.083382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.3.152.253:37061)
0925 10:17:37.083439 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.083441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.084301 (+ 860us) server_negotiation.cc:407] Connection header received
0925 10:17:37.084402 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.084403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.084414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.084443 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.086111 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.086864 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.088726 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.089396 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.089466 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.091245 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.091247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.091247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.091254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.092907 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.093412 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.093414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.093414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.093480 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.093686 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.093687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.093687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.093803 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.093935 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.094060 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.094160 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 10:17:37.094779 26545 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [term 1 FOLLOWER]: Refusing update from remote peer 0f65d5b04d0049e8809b9a0bcd993688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:37.095278 26606 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:37.096962 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f65d5b04d0049e8809b9a0bcd993688. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.097084 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:37.097982 26606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.098089 26606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: This master's current role is: LEADER
I20260925 10:17:37.098445 26592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.098537 26592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:37.098569 26615 mvcc.cc:205] Tried to move back new op lower bound from 7333197648256229376 to 7333197648211943424. Current Snapshot: MvccSnapshot[applied={T|T < 7333197648256229376}]
I20260925 10:17:37.099521 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.083498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.3.152.254:39161)
0925 10:17:37.083534 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.083536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.084109 (+ 573us) server_negotiation.cc:407] Connection header received
0925 10:17:37.084207 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.084208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.084219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.084252 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.086112 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.086853 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.087995 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.088284 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.088344 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.090290 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.090291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.090292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.090299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.092051 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.093435 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.093436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.093436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.093490 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.098855 (+ 5365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.098857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.098857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.099016 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.099124 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.099332 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.099421 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 10:17:37.100273 26612 catalog_manager.cc:1347] Generated new cluster ID: 9555f0e0cb0446398d3aa271d0146575
I20260925 10:17:37.100351 26612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 10:17:37.100698 26616 mvcc.cc:205] Tried to move back new op lower bound from 7333197648256229376 to 7333197648211943424. Current Snapshot: MvccSnapshot[applied={T|T < 7333197648256229376}]
I20260925 10:17:37.110375 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.083278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.3.152.252:37085)
0925 10:17:37.083315 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.083318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.093482 (+ 10164us) server_negotiation.cc:407] Connection header received
0925 10:17:37.093547 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.093548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.093562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.093613 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.094460 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.095180 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.097668 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.098002 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.098046 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.100078 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.100080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.100080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.100087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.101913 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.108807 (+ 6894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.108809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.108809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.108876 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.109377 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.109378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.109379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.109530 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.109626 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.110181 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.110275 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 10:17:37.110562 26496 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } attempt: 3
I20260925 10:17:37.110677 26612 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 10:17:37.111395 26612 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 10:17:37 dist-test-slave-dzwq krb5kdc[26340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790331456, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.152.253@KRBTEST.COM for kudu/127.3.152.252@KRBTEST.COM
I20260925 10:17:37.118039 26612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688: Generated new TSK 0
I20260925 10:17:37.118392 26612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 10:17:37.118754 26498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.110820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.152.252:37085 (local address 127.0.0.1:49962)
0925 10:17:37.110870 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 10:17:37.110884 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 10:17:37.110936 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 10:17:37.111432 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 10:17:37.111433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 10:17:37.111614 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 10:17:37.111669 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.111671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.112556 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.112557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.113321 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 10:17:37.113323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.113777 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.113778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 10:17:37.113837 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.114303 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 10:17:37.114308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 10:17:37.115746 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 10:17:37.117583 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.117584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.117585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.117811 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.118047 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 10:17:37.118048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 10:17:37.118048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 10:17:37.118132 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 10:17:37.118448 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 10:17:37.118449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 10:17:37.118510 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 10:17:37.118588 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 10:17:37.118685 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 10:17:37.119199 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:37.111087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.3.152.252:37085)
0925 10:17:37.111282 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.111284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.111291 (+ 7us) server_negotiation.cc:407] Connection header received
0925 10:17:37.111331 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.111331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.111341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.111425 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.111750 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.112486 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.113416 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:37.113720 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:37.113780 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:37.115836 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 10:17:37.115838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 10:17:37.115838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 10:17:37.115845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 10:17:37.117487 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.117913 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.117914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.117914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.117969 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 10:17:37.118217 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 10:17:37.118218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 10:17:37.118218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 10:17:37.118339 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 10:17:37.118423 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:37.119020 (+ 597us) server_negotiation.cc:299] Negotiation successful
0925 10:17:37.119099 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260925 10:17:37.120790 26496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: Bootstrap starting.
I20260925 10:17:37.121891 26496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: Neither blocks nor log segments found. Creating new log.
I20260925 10:17:37.122412 26496 log.cc:824] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: Log is configured to *not* fsync() on all Append() calls
I20260925 10:17:37.131526 26496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: No bootstrap required, opened a new log
I20260925 10:17:37.134075 26496 raft_consensus.cc:348] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.134249 26496 raft_consensus.cc:729] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facadad8eefa42f79b75ec04a399cbe1, State: Initialized, Role: FOLLOWER
I20260925 10:17:37.134341 26496 consensus_queue.cc:260] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } }
I20260925 10:17:37.134717 26496 sys_catalog.cc:564] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 10:17:37.134776 26469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 10:17:37.135843 26469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [term 1 FOLLOWER]: Refusing update from remote peer 0f65d5b04d0049e8809b9a0bcd993688: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 10:17:37.136430 26578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 10:17:37.136081 26470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f65d5b04d0049e8809b9a0bcd993688" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "facadad8eefa42f79b75ec04a399cbe1"
W20260925 10:17:37.136864 26635 catalog_manager.cc:1558] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 10:17:37.136914 26635 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 10:17:37.134665 26624 sys_catalog.cc:455] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.137472 26624 sys_catalog.cc:458] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:37.137547 26470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f65d5b04d0049e8809b9a0bcd993688 in term 1.
I20260925 10:17:37.138820 26637 mvcc.cc:205] Tried to move back new op lower bound from 7333197648256229376 to 7333197648211943424. Current Snapshot: MvccSnapshot[applied={T|T < 7333197648256229376}]
I20260925 10:17:37.139561 26624 sys_catalog.cc:455] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f65d5b04d0049e8809b9a0bcd993688. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.139660 26624 sys_catalog.cc:458] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 10:17:37.139703 26624 sys_catalog.cc:455] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.139750 26624 sys_catalog.cc:458] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 10:17:37.147301 26424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.145488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.3.152.254:39161)
0925 10:17:37.145535 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.145537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.146516 (+ 979us) server_negotiation.cc:407] Connection header received
0925 10:17:37.146628 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.146630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.146641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.146672 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.147214 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38330: BlockingRecv error: recv got EOF from 127.0.0.1:38330 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:17:37.148322 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.145884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.3.152.253:37061)
0925 10:17:37.145930 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.145932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.147667 (+ 1735us) server_negotiation.cc:407] Connection header received
0925 10:17:37.147773 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.147774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.147784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.147813 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.148257 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60626: BlockingRecv error: recv got EOF from 127.0.0.1:60626 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 10:17:37.149279 26576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.146222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.3.152.252:37085)
0925 10:17:37.146273 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.146275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.148671 (+ 2396us) server_negotiation.cc:407] Connection header received
0925 10:17:37.148786 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.148787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.148800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.148829 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.149222 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49976: BlockingRecv error: recv got EOF from 127.0.0.1:49976 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 10:17:37.157039 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.156316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.3.152.253:37061)
0925 10:17:37.156361 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.156364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.156481 (+ 117us) server_negotiation.cc:407] Connection header received
0925 10:17:37.156589 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.156590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.156608 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.156636 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.156985 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60642: BlockingRecv error: recv got EOF from 127.0.0.1:60642 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 10:17:37.160284 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.159456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.3.152.253:37061)
0925 10:17:37.159491 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.159494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.159753 (+ 259us) server_negotiation.cc:407] Connection header received
0925 10:17:37.159851 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.159852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.159862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.159891 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.160233 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60654: BlockingRecv error: recv got EOF from 127.0.0.1:60654 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 10:17:37.164314 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.163462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.3.152.253:37061)
0925 10:17:37.163498 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.163500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.163740 (+ 240us) server_negotiation.cc:407] Connection header received
0925 10:17:37.163835 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.163836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.163847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.163876 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.164265 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60668: BlockingRecv error: recv got EOF from 127.0.0.1:60668 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 10:17:37.170619 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.169765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.3.152.253:37061)
0925 10:17:37.169806 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.169808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.170076 (+ 268us) server_negotiation.cc:407] Connection header received
0925 10:17:37.170176 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.170177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.170189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.170220 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.170561 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60670: BlockingRecv error: recv got EOF from 127.0.0.1:60670 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:17:37.181133 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.180078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.3.152.253:37061)
0925 10:17:37.180171 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.180173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.180497 (+ 324us) server_negotiation.cc:407] Connection header received
0925 10:17:37.180601 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.180603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.180614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.180645 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.181038 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60672: BlockingRecv error: recv got EOF from 127.0.0.1:60672 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 10:17:37.199609 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.198645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.3.152.253:37061)
0925 10:17:37.198691 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.198705 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.198995 (+ 290us) server_negotiation.cc:407] Connection header received
0925 10:17:37.199119 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.199120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.199133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.199164 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.199547 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60680: BlockingRecv error: recv got EOF from 127.0.0.1:60680 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 10:17:37.234179 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.233165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.3.152.253:37061)
0925 10:17:37.233217 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.233219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.233531 (+ 312us) server_negotiation.cc:407] Connection header received
0925 10:17:37.233642 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.233643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.233655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.233687 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.234106 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60686: BlockingRecv error: recv got EOF from 127.0.0.1:60686 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 10:17:37.300678 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.299775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.3.152.253:37061)
0925 10:17:37.299844 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.299846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.300005 (+ 159us) server_negotiation.cc:407] Connection header received
0925 10:17:37.300111 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.300112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.300124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.300184 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.300612 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60690: BlockingRecv error: recv got EOF from 127.0.0.1:60690 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 10:17:37.430738 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.429988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.3.152.253:37061)
0925 10:17:37.430070 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.430072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.430161 (+ 89us) server_negotiation.cc:407] Connection header received
0925 10:17:37.430221 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.430222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.430233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.430288 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.430647 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60694: BlockingRecv error: recv got EOF from 127.0.0.1:60694 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 10:17:37.519393 26578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f65d5b04d0049e8809b9a0bcd993688" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f65d5b04d0049e8809b9a0bcd993688" member_type: VOTER last_known_addr { host: "127.3.152.254" port: 39161 } } peers { permanent_uuid: "facadad8eefa42f79b75ec04a399cbe1" member_type: VOTER last_known_addr { host: "127.3.152.253" port: 37061 } } peers { permanent_uuid: "9fd8295cd69b42aa95d523a60bbf7011" member_type: VOTER last_known_addr { host: "127.3.152.252" port: 37085 } } }
I20260925 10:17:37.519661 26578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f65d5b04d0049e8809b9a0bcd993688 [sys.catalog]: This master's current role is: LEADER
W20260925 10:17:37.688774 26500 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 10:17:37.688045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.3.152.253:37061)
0925 10:17:37.688097 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:37.688099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:37.688170 (+ 71us) server_negotiation.cc:407] Connection header received
0925 10:17:37.688250 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:37.688251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:37.688262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:37.688312 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 10:17:37.688698 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 10:17:38.082392 26603 catalog_manager.cc:1259] Loaded cluster ID: 9555f0e0cb0446398d3aa271d0146575
I20260925 10:17:38.082490 26603 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: loading cluster ID for follower catalog manager: success
I20260925 10:17:38.083395 26603 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: acquiring CA information for follower catalog manager: success
I20260925 10:17:38.083753 26603 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fd8295cd69b42aa95d523a60bbf7011: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:17:38.137355 26635 catalog_manager.cc:1259] Loaded cluster ID: 9555f0e0cb0446398d3aa271d0146575
I20260925 10:17:38.137435 26635 catalog_manager.cc:1552] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: loading cluster ID for follower catalog manager: success
I20260925 10:17:38.138276 26635 catalog_manager.cc:1574] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: acquiring CA information for follower catalog manager: success
I20260925 10:17:38.138654 26635 catalog_manager.cc:1602] T 00000000000000000000000000000000 P facadad8eefa42f79b75ec04a399cbe1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 10:17:38.204854 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:38.202088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.3.152.253:37061)
0925 10:17:38.202215 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:38.202218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:38.202358 (+ 140us) server_negotiation.cc:407] Connection header received
0925 10:17:38.202416 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:38.202417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:38.202470 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:38.202545 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:38.202718 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.203211 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.204008 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.204222 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.204276 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:38.204485 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:38.204598 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:38.204658 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:38.204708 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 10:17:38.204726 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260925 10:17:38.209525 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:38.206851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.3.152.252:37085)
0925 10:17:38.206981 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:38.206983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:38.206991 (+ 8us) server_negotiation.cc:407] Connection header received
0925 10:17:38.207032 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:38.207032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:38.207079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:38.207169 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:38.207269 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.207760 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.208671 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.208901 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.208981 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:38.209136 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:38.209264 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:38.209359 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:38.209400 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 10:17:38.209424 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 10:17:38.214736 26713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:38.211686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.3.152.252:37085)
0925 10:17:38.211729 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:38.211731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:38.211766 (+ 35us) server_negotiation.cc:407] Connection header received
0925 10:17:38.211797 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:38.211798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:38.211810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:38.211869 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:38.212215 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.212868 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.213882 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.214190 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.214228 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:38.214434 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:38.214468 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:38.214519 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:38.214611 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 10:17:38.214639 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 10:17:38.214967 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:38.211598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.3.152.254:39161)
0925 10:17:38.211710 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:38.211712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:38.211919 (+ 207us) server_negotiation.cc:407] Connection header received
0925 10:17:38.212064 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:38.212065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:38.212123 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:38.212206 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:38.212429 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.213199 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.213947 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.214164 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.214246 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:38.214447 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:38.214561 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:38.214644 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:38.214753 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 10:17:38.214782 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260925 10:17:38.215013 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 10:17:38.211816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.3.152.253:37061)
0925 10:17:38.211867 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 10:17:38.211869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 10:17:38.211942 (+ 73us) server_negotiation.cc:407] Connection header received
0925 10:17:38.212088 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 10:17:38.212089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 10:17:38.212102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 10:17:38.212161 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 10:17:38.212381 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.213147 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.214154 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 10:17:38.214453 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 10:17:38.214489 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 10:17:38.214727 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 10:17:38.214764 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 10:17:38.214800 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 10:17:38.214915 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 10:17:38.214940 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 10:17:38.216886 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 26353
I20260925 10:17:38.222734 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 26428
I20260925 10:17:38.228214 3683 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcY4CcD/build/release/bin/kudu with pid 26503
2026-09-25T10:17:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2904 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5694 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (299215 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS